Oct 30 00:37:53.477873 ip-10-0-130-171 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:37:53.896653 ip-10-0-130-171 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:37:53.896653 ip-10-0-130-171 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:37:53.896653 ip-10-0-130-171 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:37:53.896653 ip-10-0-130-171 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:37:53.904651 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904499 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.904651 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904646 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.904651 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904650 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.904651 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904654 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.904651 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904657 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.904651 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904659 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.904651 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904662 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904665 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904668 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904671 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904674 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904677 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904679 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904682 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904684 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904686 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904689 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904691 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904694 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904697 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904700 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904702 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904705 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904707 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904710 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.904849 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904712 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904719 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904722 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904726 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904730 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904733 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904736 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904738 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904740 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904743 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904745 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904748 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904750 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904752 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904756 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904758 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904761 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904764 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904766 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.905285 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904769 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904771 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904774 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904777 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904780 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904782 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904785 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904787 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904789 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904792 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904795 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904798 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904800 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904803 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904805 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904808 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904810 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904815 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904818 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904821 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.905751 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904824 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904827 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904830 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904832 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904835 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904837 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904840 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904843 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904846 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904849 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904851 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904854 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904857 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904860 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904862 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904865 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904867 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904870 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904872 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904875 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.906210 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904877 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904880 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904884 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904890 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904893 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904898 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904900 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904903 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904905 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904907 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904910 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904912 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904916 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904918 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904921 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904923 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904926 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904928 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904932 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904934 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.906691 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904937 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.904940 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905777 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905784 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905787 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905790 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905794 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905796 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905799 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905801 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905804 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905806 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905809 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905811 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905814 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905816 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905819 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905822 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905824 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905827 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.907150 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905829 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905831 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905834 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905836 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905839 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905842 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905844 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905847 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905849 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905852 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905854 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905856 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905859 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905862 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905864 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905867 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905869 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905871 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905874 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.907634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905877 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905879 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905881 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905884 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905887 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905889 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905892 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905894 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905897 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905899 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905902 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905905 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905907 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905910 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905912 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905915 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905920 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905923 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905926 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.908085 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905929 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905931 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905934 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905936 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905939 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905941 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905944 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905947 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905950 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905953 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905955 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905959 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905962 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905965 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905968 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905971 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905974 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905976 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905979 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.908653 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905982 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905984 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905987 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905989 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905992 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.905995 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906001 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906004 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906006 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906009 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906011 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906014 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906016 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906019 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906021 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906023 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906026 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906028 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906031 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906034 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.909095 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906037 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906039 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906042 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906044 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906046 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906049 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906051 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906053 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906061 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906063 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906066 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906583 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906588 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906591 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906594 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906597 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906599 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906602 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906605 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906607 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.909592 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906610 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906613 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906615 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906617 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906620 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906622 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906625 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906628 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906630 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906633 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906635 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906638 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906641 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906643 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906646 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906648 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906651 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906654 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906656 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906658 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.910061 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906662 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906665 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906668 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906672 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906675 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906678 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906680 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906683 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906685 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906688 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906691 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906693 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906696 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906698 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906701 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906704 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906706 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906708 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906711 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.910554 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906713 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906716 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906718 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906720 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906723 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906726 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906729 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906731 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906734 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906736 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906739 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906741 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906745 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906748 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906750 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906753 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906755 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906757 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906760 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.910995 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906762 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906765 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906767 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906770 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906772 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906775 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906778 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906780 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906782 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906784 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906787 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906789 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906792 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906794 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906797 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906799 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906801 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906804 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906807 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906810 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.911479 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906813 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906816 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906819 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906821 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906823 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906826 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906829 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906831 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906833 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906836 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906838 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906841 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906843 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906846 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906848 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906851 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906853 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906855 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.906859 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908270 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908289 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:37:53.911952 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908295 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908300 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908305 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908308 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908312 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908316 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908320 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908323 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908327 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908330 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908333 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908337 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908339 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908342 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908345 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908348 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908353 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908356 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908359 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908362 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908366 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908369 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908373 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:37:53.912420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908376 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908379 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908382 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908385 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908388 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908391 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908394 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908399 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908403 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908405 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908408 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908411 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908414 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908418 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908421 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908424 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908428 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908431 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908435 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908438 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908441 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908444 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908447 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908449 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908452 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:37:53.912992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908455 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908458 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908461 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908464 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908468 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908471 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908474 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908477 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908480 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908483 2577 flags.go:64] FLAG: --help="false" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908486 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-130-171.ec2.internal" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908489 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908492 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908494 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908498 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908503 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908506 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908509 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908511 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908525 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908529 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908532 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908535 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908538 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:37:53.913624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908540 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908545 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908548 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908551 2577 flags.go:64] FLAG: --lock-file="" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908554 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908557 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908560 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908566 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908569 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908572 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908575 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908578 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908581 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908584 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908587 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908592 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908595 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908599 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908602 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908604 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908607 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908610 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908613 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908616 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908619 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:37:53.914190 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908628 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908631 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908634 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908637 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908640 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908642 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908645 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908648 2577 flags.go:64] FLAG: --port="10250" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908652 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908655 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a50824c050e7716d" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908658 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908661 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908664 2577 flags.go:64] FLAG: --register-node="true" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908667 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908671 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908674 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908677 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908680 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908683 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908686 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908689 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908692 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908695 2577 flags.go:64] FLAG: --runonce="false" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908698 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908701 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908703 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:37:53.914842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908706 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908709 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908712 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908715 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908718 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908721 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908724 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908726 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908729 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908734 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908736 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908739 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908745 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908747 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908751 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908757 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908759 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908762 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908765 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908768 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908771 2577 flags.go:64] FLAG: --v="2" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908780 2577 flags.go:64] FLAG: --version="false" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908784 2577 flags.go:64] FLAG: --vmodule="" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908789 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.908792 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:37:53.915658 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908956 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908961 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908963 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908966 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908969 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908971 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908974 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908977 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908979 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908982 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908984 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908987 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908989 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908992 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908995 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.908997 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909000 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909002 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909004 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.916359 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909007 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909009 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909013 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909016 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909018 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909021 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909023 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909025 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909028 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909030 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909033 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909035 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909038 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909040 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909043 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909045 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909048 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909050 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909053 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909056 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.916992 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909058 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909060 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909063 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909065 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909067 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909070 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909072 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909075 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909078 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909080 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909083 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909085 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909088 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909091 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909095 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909098 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909101 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909103 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909106 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909108 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.917832 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909111 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909113 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909116 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909118 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909121 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909123 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909125 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909129 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909132 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909134 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909137 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909141 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909145 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909148 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909150 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909153 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909156 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909158 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909160 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.918693 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909163 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909165 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909168 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909171 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909173 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909175 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909178 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909181 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909184 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909186 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909189 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909191 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909195 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909198 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909202 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909204 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909207 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909209 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909212 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909214 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.919474 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909218 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909221 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909223 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909226 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909228 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909230 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909233 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909235 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.909274 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909334 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909340 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909343 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909346 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909348 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909351 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909354 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.920329 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909357 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909361 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909365 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909367 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909372 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909374 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909377 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909379 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909382 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909384 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909387 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909389 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909392 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909394 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909396 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909399 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909416 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909420 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909423 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.921019 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909426 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909428 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909431 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909433 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909435 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909438 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909440 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909443 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909445 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909448 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909450 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909452 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909455 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909458 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909460 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909462 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909465 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909469 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909471 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909474 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.921723 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909477 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909479 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909481 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909484 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909486 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909488 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909491 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909493 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909496 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909501 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909503 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909506 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909509 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909511 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909529 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909531 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909534 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909537 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909539 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909542 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.922389 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909544 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909547 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909549 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909552 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909554 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909557 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909561 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909563 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909566 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909570 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909572 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909575 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909577 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909579 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909582 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909584 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909587 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909589 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909592 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909594 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.923124 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909597 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909601 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909603 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909606 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909608 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909611 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909613 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909616 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909619 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909623 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909626 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909628 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909632 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909634 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909637 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909639 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909642 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909644 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909646 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.923714 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.909649 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.919567 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.919593 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.919617 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.919625 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919739 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919748 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919754 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919758 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919762 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919766 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919771 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919775 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919779 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919783 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919787 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919791 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919795 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919800 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919804 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.924164 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919808 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919812 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919816 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919828 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919832 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919836 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919840 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919844 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919849 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919852 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919856 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919861 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919864 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919869 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919874 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919878 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919882 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919887 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919890 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919895 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.924807 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919899 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919903 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919907 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919911 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919915 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919919 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919923 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919927 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919931 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919935 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919939 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919943 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919947 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919951 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919955 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919959 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919970 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919975 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919979 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919983 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.925268 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919987 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919991 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.919995 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920000 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920004 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920009 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920013 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920020 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920026 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920031 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920036 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920041 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920046 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920050 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920054 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920058 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920063 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920067 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920071 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.925744 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920077 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920083 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920088 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920092 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920097 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920101 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920105 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920109 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920113 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920117 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920129 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920134 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920138 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920142 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920146 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920150 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920155 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920159 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920164 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.926221 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920168 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920172 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920176 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920180 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920184 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920188 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920192 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920196 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920200 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920204 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920208 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920212 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920216 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.920256 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920569 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920580 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.926696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920585 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920597 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920602 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920606 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920610 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920614 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920619 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920622 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920635 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920641 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920648 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920653 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920657 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920661 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920665 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920669 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920674 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920678 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920683 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.927078 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920688 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920692 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920696 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920700 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920704 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920708 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920712 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920716 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920720 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920724 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920729 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920732 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920737 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920742 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920746 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920751 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920755 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920759 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920763 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920767 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.927646 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920771 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920775 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920787 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920791 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920795 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920799 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920803 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920807 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920811 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920816 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920820 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920824 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920829 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920833 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920837 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920841 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920845 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920849 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920853 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920857 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.928148 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920861 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920865 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920869 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920873 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920877 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920882 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920885 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920889 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920894 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920898 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920902 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920906 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920910 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920914 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.920920 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.921001 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922740 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922754 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922759 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.928766 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922764 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922770 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922775 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922780 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922785 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922790 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922794 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922799 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922810 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922815 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922819 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922824 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922828 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922833 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922846 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922859 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922880 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922885 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922890 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.929297 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922893 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922897 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922902 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922941 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922945 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.922949 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923031 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.923144 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923210 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923215 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923219 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923222 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923225 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923227 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923230 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923233 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:37:53.929754 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923235 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923238 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923241 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923243 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923247 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923250 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923252 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923255 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923257 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923260 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923262 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923265 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923268 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923271 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923274 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923276 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923279 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923282 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923284 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923287 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:37:53.930361 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923289 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923291 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923294 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923297 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923299 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923301 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923304 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923306 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923309 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923312 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923314 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923316 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923319 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923323 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923327 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923330 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923333 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923335 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923337 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923340 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:37:53.931084 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923342 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923345 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923347 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923350 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923352 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923355 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923358 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923361 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923363 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923366 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923369 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923371 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923374 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923376 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923379 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923381 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923383 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923386 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923390 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923393 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:37:53.931561 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923396 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923399 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923401 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923404 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923406 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923409 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923412 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923414 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923417 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923419 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923422 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923424 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923426 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923429 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923431 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923434 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923436 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923439 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923443 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923446 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:37:53.932141 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923448 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923450 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923453 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923455 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923457 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923460 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923463 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923465 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923468 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923471 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923473 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923475 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923478 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923480 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923482 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923485 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923487 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:53.923492 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.923601 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:37:53.932803 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.925995 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:37:53.933407 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.926993 2577 server.go:1014] "Starting client certificate rotation" Oct 30 00:37:53.933407 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.927112 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:37:53.933407 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.927158 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:37:53.933407 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.929040 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:37:53.934575 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.934552 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:37:53.946000 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.945981 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:37:53.949604 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.949588 2577 log.go:25] "Validated CRI v1 image API" Oct 30 00:37:53.950573 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.950546 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:37:53.955372 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.955346 2577 fs.go:135] Filesystem UUIDs: map[514f3dcc-b0f2-4605-96b8-a95ec06dca35:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8523c6ee-9241-4a22-8213-9b41969544ed:/dev/nvme0n1p3] Oct 30 00:37:53.955456 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.955370 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:37:53.961933 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.961780 2577 manager.go:217] Machine: {Timestamp:2026-10-30 00:37:53.960530972 +0000 UTC m=+0.355170662 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3188924 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec212968c3c1af74221e282fa2ae1902 SystemUUID:ec212968-c3c1-af74-221e-282fa2ae1902 BootID:fca3343b-3e94-4540-bff3-9f73b25346ef Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fa:69:78:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fa:69:78:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:82:a3:31:b7:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:37:53.962413 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.962399 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:37:53.962559 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.962542 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:37:53.963628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.963596 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:37:53.963789 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.963638 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-171.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:37:53.965204 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.965193 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:37:53.965238 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.965207 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:37:53.965238 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.965221 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:37:53.965238 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.965231 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:37:53.966150 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.966138 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:37:53.970121 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.970106 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:37:53.970298 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.970289 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:37:53.970951 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.970942 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:37:53.970983 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.970954 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:37:53.970983 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.970965 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:37:53.970983 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.970981 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:37:53.971098 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.970992 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:37:53.971607 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.971594 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:37:53.974345 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.974321 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:37:53.981018 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.980994 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:37:53.981855 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.981840 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:37:53.981893 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.981873 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:37:53.982052 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982039 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:37:53.982118 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982058 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:37:53.982118 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982066 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:37:53.982118 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982075 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:37:53.982118 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982082 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:37:53.982118 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982090 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:37:53.982712 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982702 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:37:53.982740 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982713 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:37:53.982740 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982720 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:37:53.982740 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982726 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:37:53.982740 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982734 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:37:53.982859 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.982744 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:37:53.983456 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.983446 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:37:53.983456 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.983456 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:37:53.987441 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.987424 2577 server.go:1263] "Started kubelet" Oct 30 00:37:53.987566 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.987475 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:37:53.987634 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.987577 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:37:53.987667 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.987654 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:37:53.988700 ip-10-0-130-171 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:37:53.989372 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.988834 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:37:53.989899 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.989887 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:37:53.994330 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:53.994306 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:37:53.995988 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.995963 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:37:53.997015 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.996998 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:37:53.997108 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:53.997091 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:53.997649 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.997632 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:37:53.997741 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.997664 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:37:53.997741 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.997634 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:37:53.997829 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.997764 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:37:53.997829 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.997772 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:37:53.997914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.997870 2577 factory.go:55] Registering systemd factory Oct 30 00:37:53.997914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.997883 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:37:53.998081 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.998069 2577 factory.go:153] Registering CRI-O factory Oct 30 00:37:53.998121 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.998083 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 00:37:53.998150 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.998122 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:37:53.998150 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.998136 2577 factory.go:103] Registering Raw factory Oct 30 00:37:53.998150 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.998146 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 00:37:53.999413 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:53.999391 2577 manager.go:319] Starting recovery of all containers Oct 30 00:37:54.010465 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.010447 2577 manager.go:324] Recovery completed Oct 30 00:37:54.012575 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.012357 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 00:37:54.012676 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.012656 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-171.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:37:54.013218 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.012356 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-171.ec2.internal.18e327a6cd61f69f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-171.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-171.ec2.internal,},FirstTimestamp:2026-10-30 00:37:53.987393183 +0000 UTC m=+0.382032826,LastTimestamp:2026-10-30 00:37:53.987393183 +0000 UTC m=+0.382032826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-171.ec2.internal,}" Oct 30 00:37:54.016258 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.016241 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:54.018380 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.018356 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:54.018457 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.018401 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:54.018457 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.018416 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:54.019055 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.019041 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:37:54.019055 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.019054 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:37:54.019125 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.019071 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:37:54.019125 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.019086 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:54.019785 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.019769 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:54.019840 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.019801 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:54.019840 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.019814 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:54.021328 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.021314 2577 policy_none.go:50] "Start" Oct 30 00:37:54.021363 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.021336 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:37:54.021363 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.021354 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:37:54.022438 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.022420 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:37:54.023412 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.023400 2577 policy_none.go:44] "Start" Oct 30 00:37:54.032427 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.032399 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 00:37:54.032427 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.032404 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 00:37:54.032573 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.032456 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 00:37:54.041045 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.041021 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-422kx" Oct 30 00:37:54.048899 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.048874 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-422kx" Oct 30 00:37:54.054543 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.053916 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.059219 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.059269 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.059283 2577 server.go:87] "Starting device plugin registration server" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.059691 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.059704 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.059918 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.059992 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.059999 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.061012 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.061066 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.061153 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-171.ec2.internal.18e327a6cf3acb1d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-171.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-171.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-171.ec2.internal,},FirstTimestamp:2026-10-30 00:37:54.018380573 +0000 UTC m=+0.413020224,LastTimestamp:2026-10-30 00:37:54.018380573 +0000 UTC m=+0.413020224,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-171.ec2.internal,}" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.074607 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.075970 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.075991 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.076009 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:37:54.084163 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.076074 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:37:54.090360 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.090342 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:37:54.160606 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.160506 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:54.161506 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.161480 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:54.161506 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.161510 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:54.161674 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.161538 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:54.161674 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.161565 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.176453 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.176423 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal"] Oct 30 00:37:54.176599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.176495 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:54.177399 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.177372 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:54.177493 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.177405 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:54.177493 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.177420 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:54.179510 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.179496 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:54.179654 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.179638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.179707 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.179674 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:54.180307 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.180291 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:54.180363 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.180314 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:54.180363 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.180325 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:54.180363 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.180335 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:54.180363 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.180352 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:54.180363 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.180362 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:54.181429 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.181414 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.181509 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.181433 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-171.ec2.internal\": node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.182337 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.182322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.182414 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.182355 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:54.183124 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.183108 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:54.183217 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.183139 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:54.183217 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.183152 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:54.198974 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.198945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.199135 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.199021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.199135 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.199052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92d13e825dfc8d10deb63823b2be0da4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" (UID: \"92d13e825dfc8d10deb63823b2be0da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.211251 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.211225 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-171.ec2.internal\" not found" node="ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.214183 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.214163 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-171.ec2.internal\" not found" node="ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.216840 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.216821 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.299197 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.299167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.299197 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.299196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.299419 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.299214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92d13e825dfc8d10deb63823b2be0da4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" (UID: \"92d13e825dfc8d10deb63823b2be0da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.299419 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.299251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92d13e825dfc8d10deb63823b2be0da4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" (UID: \"92d13e825dfc8d10deb63823b2be0da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.299419 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.299270 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.299419 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.299274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.317558 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.317509 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.418370 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.418287 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.515276 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.515231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.518320 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.518297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:37:54.518413 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.518372 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.618838 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.618798 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.719399 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.719306 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.819911 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.819870 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.920392 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:54.920358 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:54.926544 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.926505 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:37:54.926731 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:54.926713 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:37:55.021165 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:55.021089 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:55.054491 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.054439 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:32:54 +0000 UTC" deadline="2028-04-26 21:13:20.142434321 +0000 UTC" Oct 30 00:37:55.054491 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.054482 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13076h35m25.087956092s" Oct 30 00:37:55.068864 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:55.068828 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d13e825dfc8d10deb63823b2be0da4.slice/crio-30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488 WatchSource:0}: Error finding container 30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488: Status 404 returned error can't find the container with id 30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488 Oct 30 00:37:55.069499 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:55.069476 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11e125d29b156d087bacedc1ce8a773.slice/crio-49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a WatchSource:0}: Error finding container 49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a: Status 404 returned error can't find the container with id 49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a Oct 30 00:37:55.073792 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.073775 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:37:55.081833 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.081762 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" event={"ID":"92d13e825dfc8d10deb63823b2be0da4","Type":"ContainerStarted","Data":"30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488"} Oct 30 00:37:55.082721 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.082701 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" event={"ID":"e11e125d29b156d087bacedc1ce8a773","Type":"ContainerStarted","Data":"49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a"} Oct 30 00:37:55.121338 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:55.121299 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:55.158413 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.158385 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:37:55.159619 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.159596 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:37:55.159735 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.159636 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:37:55.159735 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.159648 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:37:55.222187 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:55.222150 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:55.322826 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:55.322744 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:37:55.398772 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.398743 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:37:55.427724 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.427692 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:37:55.498108 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.498072 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:37:55.519445 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.519405 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:37:55.520399 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.520378 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:37:55.544467 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.544434 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:37:55.642172 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.642086 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:37:55.784850 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.784812 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:37:55.972390 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.972294 2577 apiserver.go:52] "Watching apiserver" Oct 30 00:37:55.997361 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:55.997316 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:37:56.004191 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.004159 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:37:56.004619 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.004587 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-bzs4r","openshift-network-diagnostics/network-check-target-qk7xr","kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal","openshift-dns/node-resolver-qnlzg","openshift-network-operator/iptables-alerter-4lhrd","kube-system/global-pull-secret-syncer-vctgv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx","openshift-cluster-node-tuning-operator/tuned-k7g8f","openshift-image-registry/node-ca-2j4gq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-wmvs5","kube-system/konnectivity-agent-8f8v5","openshift-multus/multus-2lt89","openshift-multus/network-metrics-daemon-xwnkz"] Oct 30 00:37:56.007666 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.007629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.011287 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.011261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.014154 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.014131 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:56.014273 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.014218 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:37:56.016696 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.016672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.019235 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.019212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.021844 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.021823 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.022001 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.021896 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:37:56.024855 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.024831 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:37:56.024855 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.024847 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:37:56.025025 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.024956 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:37:56.025025 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.024973 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t5wsq\"" Oct 30 00:37:56.025128 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.025115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.027712 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.027566 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:37:56.027712 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.027615 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:37:56.027712 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.027658 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:37:56.027712 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.027666 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tmln4\"" Oct 30 00:37:56.027712 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.027706 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:37:56.028198 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.028177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.030526 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.030504 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:37:56.030634 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.030607 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:37:56.033747 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.033726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.036763 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.036744 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.039469 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.039448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.050807 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.050775 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:37:56.050946 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.050864 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mgptt" Oct 30 00:37:56.098799 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.098770 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:37:56.109747 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.109698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.109977 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.109958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-multus\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.110123 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd14a22e-0b4d-45eb-a25e-111817cfe34d-hosts-file\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.110276 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-conf\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.110276 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-systemd\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.110276 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd14a22e-0b4d-45eb-a25e-111817cfe34d-tmp-dir\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.110276 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110225 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3561da-40f3-40db-a14b-ad558c5236d7-agent-certs\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.110276 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzqh\" (UniqueName: \"kubernetes.io/projected/8c300dc5-864d-4faf-ad58-807fb3813895-kube-api-access-bgzqh\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.110494 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-systemd-units\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110494 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-netns\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110494 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-etc-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110494 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-kubernetes\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.110494 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-tmp\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.110494 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbd7g\" (UniqueName: \"kubernetes.io/projected/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-kube-api-access-cbd7g\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110720 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-daemon-config\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.110720 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-device-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.110720 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.110720 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-systemd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110720 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-ovn\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110720 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovn-node-metrics-cert\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110720 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:56.110949 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7849b028-2379-4a22-bee9-d1fa3544b3b4-iptables-alerter-script\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.110949 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtht\" (UniqueName: \"kubernetes.io/projected/7849b028-2379-4a22-bee9-d1fa3544b3b4-kube-api-access-fvtht\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.110949 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-script-lib\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.110949 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-bin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.110949 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-multus-certs\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.110949 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjtf\" (UniqueName: \"kubernetes.io/projected/2b542409-f192-467d-92a8-626445065e1b-kube-api-access-xgjtf\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.110949 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-lib-modules\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-node-log\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.110989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-hostroot\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-serviceca\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-registration-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-sys-fs\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2vg\" (UniqueName: \"kubernetes.io/projected/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-kube-api-access-8j2vg\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.111184 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-etc-tuned\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-var-lib-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-etc-kubernetes\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-kubelet\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-slash\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qbk\" (UniqueName: \"kubernetes.io/projected/dd14a22e-0b4d-45eb-a25e-111817cfe34d-kube-api-access-z6qbk\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-dbus\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.111499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3561da-40f3-40db-a14b-ad558c5236d7-konnectivity-ca\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-system-cni-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-run\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-netd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-socket-dir-parent\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-host\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.111800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-cnibin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-k8s-cni-cncf-io\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-kubelet-config\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-socket-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-sys\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-kubelet\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.111994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-host\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-etc-selinux\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-env-overrides\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-conf-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.112158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8d8n\" (UniqueName: \"kubernetes.io/projected/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-kube-api-access-q8d8n\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.112709 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.112709 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-cni-binary-copy\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.112848 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-bin\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.112848 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-config\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.112848 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggc6f\" (UniqueName: \"kubernetes.io/projected/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kube-api-access-ggc6f\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.112848 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:37:56.112985 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-var-lib-kubelet\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.112985 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7849b028-2379-4a22-bee9-d1fa3544b3b4-host-slash\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.112985 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-modprobe-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.112985 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysconfig\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.113164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.112989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-log-socket\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.113164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.113017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-system-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.113164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.113045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-netns\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.113164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.113075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrsc4\" (UniqueName: \"kubernetes.io/projected/7997439e-9f1e-495d-bac8-6674d4f4a48a-kube-api-access-lrsc4\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.113164 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.113131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-cnibin\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.134948 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.134902 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q2s6t\"" Oct 30 00:37:56.135158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.134991 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:37:56.135158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135070 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:37:56.135269 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135174 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:37:56.135324 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135308 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:37:56.135422 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135407 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k8zgn\"" Oct 30 00:37:56.135470 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135454 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:37:56.135569 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135522 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:37:56.135697 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135664 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:37:56.135751 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135742 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ckcc7\"" Oct 30 00:37:56.135965 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.135946 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:37:56.136268 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136241 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg5gj\"" Oct 30 00:37:56.136350 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136312 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:37:56.136350 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136338 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:37:56.136439 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136360 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:37:56.136439 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136425 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:37:56.136560 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136465 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:37:56.136560 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136538 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-545pj\"" Oct 30 00:37:56.136646 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136626 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:37:56.136707 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136693 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:37:56.136786 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136752 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5s9q7\"" Oct 30 00:37:56.136842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136785 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:37:56.136842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136830 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:37:56.136935 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136879 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:37:56.136935 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136917 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bp5j\"" Oct 30 00:37:56.137025 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136949 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:37:56.137025 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.136985 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:37:56.213736 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3561da-40f3-40db-a14b-ad558c5236d7-agent-certs\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.213909 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzqh\" (UniqueName: \"kubernetes.io/projected/8c300dc5-864d-4faf-ad58-807fb3813895-kube-api-access-bgzqh\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.213909 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-systemd-units\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.213909 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-netns\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.213909 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-etc-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.213909 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-kubernetes\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.213909 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-tmp\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.213909 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbd7g\" (UniqueName: \"kubernetes.io/projected/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-kube-api-access-cbd7g\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-daemon-config\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-device-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.213975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-systemd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-ovn\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovn-node-metrics-cert\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7849b028-2379-4a22-bee9-d1fa3544b3b4-iptables-alerter-script\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtht\" (UniqueName: \"kubernetes.io/projected/7849b028-2379-4a22-bee9-d1fa3544b3b4-kube-api-access-fvtht\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-script-lib\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-bin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-multus-certs\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjtf\" (UniqueName: \"kubernetes.io/projected/2b542409-f192-467d-92a8-626445065e1b-kube-api-access-xgjtf\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-lib-modules\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-node-log\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-hostroot\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.214318 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-serviceca\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-registration-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-sys-fs\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2vg\" (UniqueName: \"kubernetes.io/projected/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-kube-api-access-8j2vg\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-etc-tuned\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-var-lib-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-etc-kubernetes\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-kubelet\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-slash\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qbk\" (UniqueName: \"kubernetes.io/projected/dd14a22e-0b4d-45eb-a25e-111817cfe34d-kube-api-access-z6qbk\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-dbus\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3561da-40f3-40db-a14b-ad558c5236d7-konnectivity-ca\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-system-cni-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-run\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.215004 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-netd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-socket-dir-parent\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-host\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-cnibin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-k8s-cni-cncf-io\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.214988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-kubelet-config\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-socket-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-sys\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215091 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-kubelet\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-host\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-etc-selinux\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-env-overrides\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.215638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-conf-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8d8n\" (UniqueName: \"kubernetes.io/projected/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-kube-api-access-q8d8n\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-cni-binary-copy\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-bin\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-config\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggc6f\" (UniqueName: \"kubernetes.io/projected/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kube-api-access-ggc6f\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-var-lib-kubelet\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7849b028-2379-4a22-bee9-d1fa3544b3b4-host-slash\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-modprobe-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysconfig\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-log-socket\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-system-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-netns\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrsc4\" (UniqueName: \"kubernetes.io/projected/7997439e-9f1e-495d-bac8-6674d4f4a48a-kube-api-access-lrsc4\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.216247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.215717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-cnibin\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.216936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.216255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.216936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.216286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-multus\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.216936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.216328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd14a22e-0b4d-45eb-a25e-111817cfe34d-hosts-file\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.216936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.216357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-conf\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.216936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.216386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-systemd\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.216936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.216413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd14a22e-0b4d-45eb-a25e-111817cfe34d-tmp-dir\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.216936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.216862 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd14a22e-0b4d-45eb-a25e-111817cfe34d-tmp-dir\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.217238 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217219 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:37:56.217433 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217401 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-run\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.217664 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.217734 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217694 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-netd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.217782 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217739 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.217819 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217791 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-socket-dir-parent\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.217854 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217816 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-systemd-units\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.217887 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.217860 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:56.217887 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217872 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-netns\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.217947 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-etc-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.217947 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.217931 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret podName:447c7c04-4097-4065-9e32-ac6b6dc15706 nodeName:}" failed. No retries permitted until 2026-10-30 00:37:56.717896996 +0000 UTC m=+3.112536631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret") pod "global-pull-secret-syncer-vctgv" (UID: "447c7c04-4097-4065-9e32-ac6b6dc15706") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:56.218026 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.217991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-kubernetes\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.218129 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218107 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.218373 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218268 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-socket-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.218464 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218445 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.218522 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218499 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-sys\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.218522 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218503 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-host\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.218646 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218537 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-cnibin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.218646 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-kubelet\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.218646 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218590 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-k8s-cni-cncf-io\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.218646 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-host\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.219010 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-kubelet-config\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.219010 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218700 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-etc-selinux\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.219010 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-log-socket\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.219010 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.218825 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.219224 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.219148 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-env-overrides\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.219224 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.219191 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-bin\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.219721 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.219407 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-hostroot\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.219721 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.219545 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-var-lib-kubelet\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.219721 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.219581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-system-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.219721 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.219629 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-netns\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.219721 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.219647 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:56.220023 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.219823 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:37:56.719791833 +0000 UTC m=+3.114431486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:56.220091 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220035 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-cnibin\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.220151 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220091 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.220151 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220141 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-multus\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.220254 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-modprobe-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.220254 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd14a22e-0b4d-45eb-a25e-111817cfe34d-hosts-file\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.220254 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7849b028-2379-4a22-bee9-d1fa3544b3b4-host-slash\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.220389 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-multus-certs\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.220389 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220325 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-systemd\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.220389 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220338 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-conf\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-conf-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-dbus\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-systemd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-node-log\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220778 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-device-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-etc-kubernetes\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220981 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-kubelet\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221037 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-sys-fs\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-script-lib\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-bin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221133 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-cni-binary-copy\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221161 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-lib-modules\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221402 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221496 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-config\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.220980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-slash\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221728 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-ovn\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.221863 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-system-cni-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.222959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.221967 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysconfig\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.222959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.222007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-registration-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.222959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.222061 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-var-lib-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.222959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.222239 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.227170 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.227141 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-serviceca\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.227307 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.227158 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-daemon-config\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.227307 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.227202 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.227672 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.227635 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.228165 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.228144 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-tmp\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.228500 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.228477 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3561da-40f3-40db-a14b-ad558c5236d7-konnectivity-ca\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.228782 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.228757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3561da-40f3-40db-a14b-ad558c5236d7-agent-certs\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.229035 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.228862 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7849b028-2379-4a22-bee9-d1fa3544b3b4-iptables-alerter-script\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.230629 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.230602 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-etc-tuned\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.232416 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.232395 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovn-node-metrics-cert\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.238939 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.238914 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:37:56.238939 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.238942 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:37:56.239112 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.238955 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gplz4 for pod openshift-network-diagnostics/network-check-target-qk7xr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:37:56.241795 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.241762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzqh\" (UniqueName: \"kubernetes.io/projected/8c300dc5-864d-4faf-ad58-807fb3813895-kube-api-access-bgzqh\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.247108 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.246938 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbd7g\" (UniqueName: \"kubernetes.io/projected/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-kube-api-access-cbd7g\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.247108 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.247047 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggc6f\" (UniqueName: \"kubernetes.io/projected/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kube-api-access-ggc6f\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.247927 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.247901 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8d8n\" (UniqueName: \"kubernetes.io/projected/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-kube-api-access-q8d8n\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.249876 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.248779 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrsc4\" (UniqueName: \"kubernetes.io/projected/7997439e-9f1e-495d-bac8-6674d4f4a48a-kube-api-access-lrsc4\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.249876 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.249381 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4 podName:0007937d-ef02-42e5-84de-6d9b8706537d nodeName:}" failed. No retries permitted until 2026-10-30 00:37:56.739016547 +0000 UTC m=+3.133656203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gplz4" (UniqueName: "kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4") pod "network-check-target-qk7xr" (UID: "0007937d-ef02-42e5-84de-6d9b8706537d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:37:56.249876 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.249439 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjtf\" (UniqueName: \"kubernetes.io/projected/2b542409-f192-467d-92a8-626445065e1b-kube-api-access-xgjtf\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.249876 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.249470 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qbk\" (UniqueName: \"kubernetes.io/projected/dd14a22e-0b4d-45eb-a25e-111817cfe34d-kube-api-access-z6qbk\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.249876 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.249474 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtht\" (UniqueName: \"kubernetes.io/projected/7849b028-2379-4a22-bee9-d1fa3544b3b4-kube-api-access-fvtht\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.251133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.251105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2vg\" (UniqueName: \"kubernetes.io/projected/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-kube-api-access-8j2vg\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:56.329749 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.329701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:37:56.332691 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.332463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2lt89" Oct 30 00:37:56.343475 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.343448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:37:56.350233 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.350209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:37:56.354634 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:56.354561 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd14a22e_0b4d_45eb_a25e_111817cfe34d.slice/crio-f808938b92d09766bb6ef08c34f05523b6576dbd13a917a0b10ff06913c7ac1b WatchSource:0}: Error finding container f808938b92d09766bb6ef08c34f05523b6576dbd13a917a0b10ff06913c7ac1b: Status 404 returned error can't find the container with id f808938b92d09766bb6ef08c34f05523b6576dbd13a917a0b10ff06913c7ac1b Oct 30 00:37:56.358569 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.358402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:37:56.363059 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:56.362997 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7849b028_2379_4a22_bee9_d1fa3544b3b4.slice/crio-8dd74a23ea8e383175471708ecbcd5b636e23fce31c8fa8c89fcac3e6976d91f WatchSource:0}: Error finding container 8dd74a23ea8e383175471708ecbcd5b636e23fce31c8fa8c89fcac3e6976d91f: Status 404 returned error can't find the container with id 8dd74a23ea8e383175471708ecbcd5b636e23fce31c8fa8c89fcac3e6976d91f Oct 30 00:37:56.371470 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:56.371436 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aa2cb7_11fc_4a3e_8bad_519c2c0abcbb.slice/crio-818ed041536d1539c25505ddee02ccd10a8a9d134ccd71fbb5f4db088bb66a7f WatchSource:0}: Error finding container 818ed041536d1539c25505ddee02ccd10a8a9d134ccd71fbb5f4db088bb66a7f: Status 404 returned error can't find the container with id 818ed041536d1539c25505ddee02ccd10a8a9d134ccd71fbb5f4db088bb66a7f Oct 30 00:37:56.379245 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.379218 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:37:56.386912 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.386718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:37:56.391187 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:56.391154 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c300dc5_864d_4faf_ad58_807fb3813895.slice/crio-14a28eea12639a632d0b50550c2f8a7763a613a4d598779e39e71002e3a2b360 WatchSource:0}: Error finding container 14a28eea12639a632d0b50550c2f8a7763a613a4d598779e39e71002e3a2b360: Status 404 returned error can't find the container with id 14a28eea12639a632d0b50550c2f8a7763a613a4d598779e39e71002e3a2b360 Oct 30 00:37:56.397655 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:56.397610 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b542409_f192_467d_92a8_626445065e1b.slice/crio-e1a017f0554efd71a70812367c9a628df56e8782c334fbecb1c14f27d2820449 WatchSource:0}: Error finding container e1a017f0554efd71a70812367c9a628df56e8782c334fbecb1c14f27d2820449: Status 404 returned error can't find the container with id e1a017f0554efd71a70812367c9a628df56e8782c334fbecb1c14f27d2820449 Oct 30 00:37:56.398412 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.398389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:37:56.405005 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.404970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:37:56.409990 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:56.409960 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3561da_40f3_40db_a14b_ad558c5236d7.slice/crio-9ad6334276b714b82b4942ebbbf441cf4fcbe8a209faa05249d1ea0d06d24bd1 WatchSource:0}: Error finding container 9ad6334276b714b82b4942ebbbf441cf4fcbe8a209faa05249d1ea0d06d24bd1: Status 404 returned error can't find the container with id 9ad6334276b714b82b4942ebbbf441cf4fcbe8a209faa05249d1ea0d06d24bd1 Oct 30 00:37:56.418146 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:37:56.418115 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e3d552_3b9e_4c1e_bc5c_b63204d1ce6c.slice/crio-807a8c559e2d5f2c65dc80104769fe6a67023faaed1326f157148c06c34d59ea WatchSource:0}: Error finding container 807a8c559e2d5f2c65dc80104769fe6a67023faaed1326f157148c06c34d59ea: Status 404 returned error can't find the container with id 807a8c559e2d5f2c65dc80104769fe6a67023faaed1326f157148c06c34d59ea Oct 30 00:37:56.720557 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.720272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:56.720730 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.720615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:56.720730 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.720633 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:56.720730 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.720703 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:37:57.720686163 +0000 UTC m=+4.115325793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:56.720881 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.720733 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:56.720881 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.720792 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret podName:447c7c04-4097-4065-9e32-ac6b6dc15706 nodeName:}" failed. No retries permitted until 2026-10-30 00:37:57.720775735 +0000 UTC m=+4.115415383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret") pod "global-pull-secret-syncer-vctgv" (UID: "447c7c04-4097-4065-9e32-ac6b6dc15706") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:56.822152 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:56.822081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:37:56.822298 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.822280 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:37:56.822355 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.822302 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:37:56.822355 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.822316 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gplz4 for pod openshift-network-diagnostics/network-check-target-qk7xr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:37:56.822458 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:56.822383 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4 podName:0007937d-ef02-42e5-84de-6d9b8706537d nodeName:}" failed. No retries permitted until 2026-10-30 00:37:57.822363848 +0000 UTC m=+4.217003501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gplz4" (UniqueName: "kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4") pod "network-check-target-qk7xr" (UID: "0007937d-ef02-42e5-84de-6d9b8706537d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:37:57.054538 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.054329 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:32:56 +0000 UTC" deadline="2028-05-16 14:19:38.420755029 +0000 UTC" Oct 30 00:37:57.054538 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.054396 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13549h41m41.366363847s" Oct 30 00:37:57.077453 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.076873 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:57.077453 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.077037 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:37:57.092627 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.092565 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" event={"ID":"92d13e825dfc8d10deb63823b2be0da4","Type":"ContainerStarted","Data":"576dfca443b02c7019b1553b2560df41925686d3e5a644bad32038461de7896c"} Oct 30 00:37:57.097957 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.097917 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8f8v5" event={"ID":"ee3561da-40f3-40db-a14b-ad558c5236d7","Type":"ContainerStarted","Data":"9ad6334276b714b82b4942ebbbf441cf4fcbe8a209faa05249d1ea0d06d24bd1"} Oct 30 00:37:57.101439 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.101395 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerStarted","Data":"e1a017f0554efd71a70812367c9a628df56e8782c334fbecb1c14f27d2820449"} Oct 30 00:37:57.105987 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.105885 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2j4gq" event={"ID":"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb","Type":"ContainerStarted","Data":"818ed041536d1539c25505ddee02ccd10a8a9d134ccd71fbb5f4db088bb66a7f"} Oct 30 00:37:57.110537 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.110499 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" event={"ID":"08aeeb89-9b10-41a0-a8bf-d07d7642ff66","Type":"ContainerStarted","Data":"ab08e824cc7eeb6566146f9c4cc49fbe3bddde04c1419f27fc4def652e5cd551"} Oct 30 00:37:57.124016 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.123972 2577 generic.go:356] "Generic (PLEG): container finished" podID="e11e125d29b156d087bacedc1ce8a773" containerID="a158535e6e4f5b602cf7a0648e547c542e83fc3767a3497ed2c879b299ddc1f9" exitCode=0 Oct 30 00:37:57.124207 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.124087 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" event={"ID":"e11e125d29b156d087bacedc1ce8a773","Type":"ContainerDied","Data":"a158535e6e4f5b602cf7a0648e547c542e83fc3767a3497ed2c879b299ddc1f9"} Oct 30 00:37:57.128468 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.128423 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"807a8c559e2d5f2c65dc80104769fe6a67023faaed1326f157148c06c34d59ea"} Oct 30 00:37:57.136002 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.135848 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" event={"ID":"8c300dc5-864d-4faf-ad58-807fb3813895","Type":"ContainerStarted","Data":"14a28eea12639a632d0b50550c2f8a7763a613a4d598779e39e71002e3a2b360"} Oct 30 00:37:57.136002 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.135944 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" podStartSLOduration=1.135923691 podStartE2EDuration="1.135923691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:57.1348015 +0000 UTC m=+3.529441155" watchObservedRunningTime="2026-10-30 00:37:57.135923691 +0000 UTC m=+3.530563347" Oct 30 00:37:57.142214 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.142119 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4lhrd" event={"ID":"7849b028-2379-4a22-bee9-d1fa3544b3b4","Type":"ContainerStarted","Data":"8dd74a23ea8e383175471708ecbcd5b636e23fce31c8fa8c89fcac3e6976d91f"} Oct 30 00:37:57.144033 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.143994 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnlzg" event={"ID":"dd14a22e-0b4d-45eb-a25e-111817cfe34d","Type":"ContainerStarted","Data":"f808938b92d09766bb6ef08c34f05523b6576dbd13a917a0b10ff06913c7ac1b"} Oct 30 00:37:57.154839 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.154802 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2lt89" event={"ID":"7997439e-9f1e-495d-bac8-6674d4f4a48a","Type":"ContainerStarted","Data":"3114d7b1f54d710ef44bf0c9fb469ca25f4d481e887be9f94d78e66b3d887044"} Oct 30 00:37:57.730027 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.729984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:57.730227 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.730082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:57.730289 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.730230 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:57.730338 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.730293 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:37:59.730275731 +0000 UTC m=+6.124915377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:57.730792 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.730754 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:57.730887 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.730821 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret podName:447c7c04-4097-4065-9e32-ac6b6dc15706 nodeName:}" failed. No retries permitted until 2026-10-30 00:37:59.730803467 +0000 UTC m=+6.125443113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret") pod "global-pull-secret-syncer-vctgv" (UID: "447c7c04-4097-4065-9e32-ac6b6dc15706") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:57.830616 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:57.830539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:37:57.830798 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.830750 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:37:57.830798 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.830780 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:37:57.830798 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.830796 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gplz4 for pod openshift-network-diagnostics/network-check-target-qk7xr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:37:57.830962 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:57.830860 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4 podName:0007937d-ef02-42e5-84de-6d9b8706537d nodeName:}" failed. No retries permitted until 2026-10-30 00:37:59.830839288 +0000 UTC m=+6.225478923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gplz4" (UniqueName: "kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4") pod "network-check-target-qk7xr" (UID: "0007937d-ef02-42e5-84de-6d9b8706537d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:37:58.054780 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:58.054603 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:32:56 +0000 UTC" deadline="2028-07-10 08:25:52.828224778 +0000 UTC" Oct 30 00:37:58.054780 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:58.054686 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14863h47m54.773542877s" Oct 30 00:37:58.080161 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:58.079394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:58.080161 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:58.079530 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:37:58.080161 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:58.079994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:37:58.080161 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:58.080096 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:37:58.164847 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:58.164751 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" event={"ID":"e11e125d29b156d087bacedc1ce8a773","Type":"ContainerStarted","Data":"090a7cd61050c136a0e914777c01afb91a2c4f690f058a499ac1159b3b8f14fd"} Oct 30 00:37:59.077663 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:59.076832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:59.077663 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.076977 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:37:59.182959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:59.182872 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerStarted","Data":"f5f85dcf83718dd6254bf0e659578ad54b5dc3e1e9168c78cf87d8b9a247d628"} Oct 30 00:37:59.259686 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:59.259621 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" podStartSLOduration=3.259580561 podStartE2EDuration="3.259580561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:58.20865615 +0000 UTC m=+4.603295806" watchObservedRunningTime="2026-10-30 00:37:59.259580561 +0000 UTC m=+5.654220218" Oct 30 00:37:59.748679 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:59.747855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:37:59.748679 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:59.747932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:37:59.748679 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.748078 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:59.748679 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.748148 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret podName:447c7c04-4097-4065-9e32-ac6b6dc15706 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:03.748132775 +0000 UTC m=+10.142772432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret") pod "global-pull-secret-syncer-vctgv" (UID: "447c7c04-4097-4065-9e32-ac6b6dc15706") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:37:59.748679 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.748215 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:59.748679 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.748243 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:03.748234837 +0000 UTC m=+10.142874480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:37:59.849402 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:37:59.848705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:37:59.849402 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.848889 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:37:59.849402 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.848908 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:37:59.849402 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.848921 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gplz4 for pod openshift-network-diagnostics/network-check-target-qk7xr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:37:59.849402 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:37:59.848987 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4 podName:0007937d-ef02-42e5-84de-6d9b8706537d nodeName:}" failed. No retries permitted until 2026-10-30 00:38:03.848969395 +0000 UTC m=+10.243609043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gplz4" (UniqueName: "kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4") pod "network-check-target-qk7xr" (UID: "0007937d-ef02-42e5-84de-6d9b8706537d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:38:00.077129 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:00.077041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:00.077301 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:00.077185 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:00.084599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:00.076995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:00.084599 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:00.084042 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:00.190341 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:00.190292 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="f5f85dcf83718dd6254bf0e659578ad54b5dc3e1e9168c78cf87d8b9a247d628" exitCode=0 Oct 30 00:38:00.190553 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:00.190362 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerDied","Data":"f5f85dcf83718dd6254bf0e659578ad54b5dc3e1e9168c78cf87d8b9a247d628"} Oct 30 00:38:01.077357 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:01.076768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:01.077357 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:01.076938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:01.203588 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:01.202846 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8f8v5" event={"ID":"ee3561da-40f3-40db-a14b-ad558c5236d7","Type":"ContainerStarted","Data":"4d4532e7743316ba7dcb9170fdcfdbcc956ce50f1fe875f77ea8aa5a64e7497f"} Oct 30 00:38:02.079675 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.079122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:02.079675 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:02.079264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:02.079675 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.079345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:02.079675 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:02.079418 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:02.216359 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.216308 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2j4gq" event={"ID":"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb","Type":"ContainerStarted","Data":"39422adc4e450ad46d11ad058f5120b6deb3196ca35f36c0e9c67787dcb92238"} Oct 30 00:38:02.221866 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.221827 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" event={"ID":"08aeeb89-9b10-41a0-a8bf-d07d7642ff66","Type":"ContainerStarted","Data":"c4b2ea9a547dabec80567e9aa871528b0efef549a1cdad2bce88974d3e246ab9"} Oct 30 00:38:02.259674 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.259581 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8f8v5" podStartSLOduration=3.283890958 podStartE2EDuration="7.259557917s" totalImagesPullingTime="3.975666959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:01.245297322 +0000 UTC m=+7.639936977" watchObservedRunningTime="2026-10-30 00:38:02.259557917 +0000 UTC m=+8.654197572" Oct 30 00:38:02.355715 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.355626 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:02.356747 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.356502 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:02.400489 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:02.399531 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2j4gq" podStartSLOduration=2.335017028 podStartE2EDuration="7.39951235s" totalImagesPullingTime="5.064495322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:02.26029961 +0000 UTC m=+8.654939265" watchObservedRunningTime="2026-10-30 00:38:02.39951235 +0000 UTC m=+8.794152004" Oct 30 00:38:03.077281 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:03.077231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:03.077496 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.077392 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:03.227645 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:03.227302 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:03.233115 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:03.233078 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:03.616624 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:03.616593 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:38:03.786652 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:03.786531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:03.786652 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:03.786615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:03.786880 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.786777 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:38:03.786880 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.786843 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret podName:447c7c04-4097-4065-9e32-ac6b6dc15706 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:11.786824848 +0000 UTC m=+18.181464484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret") pod "global-pull-secret-syncer-vctgv" (UID: "447c7c04-4097-4065-9e32-ac6b6dc15706") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:38:03.787366 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.787298 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:38:03.787366 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.787355 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:11.787338372 +0000 UTC m=+18.181978006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:38:03.887245 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:03.887199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:03.887462 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.887436 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:38:03.887535 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.887466 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:38:03.887535 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.887480 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gplz4 for pod openshift-network-diagnostics/network-check-target-qk7xr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:38:03.887673 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:03.887546 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4 podName:0007937d-ef02-42e5-84de-6d9b8706537d nodeName:}" failed. No retries permitted until 2026-10-30 00:38:11.887525666 +0000 UTC m=+18.282165301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gplz4" (UniqueName: "kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4") pod "network-check-target-qk7xr" (UID: "0007937d-ef02-42e5-84de-6d9b8706537d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:38:04.067002 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:04.066123 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:38:03.616618246Z","UUID":"9b5f6e60-bc1d-4966-985e-c751d5f44f24","Handler":null,"Name":"","Endpoint":""} Oct 30 00:38:04.068245 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:04.067964 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:38:04.068245 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:04.067997 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:38:04.077888 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:04.077847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:04.078072 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:04.077994 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:04.079148 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:04.079097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:04.079272 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:04.079199 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:04.232433 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:04.232390 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" event={"ID":"08aeeb89-9b10-41a0-a8bf-d07d7642ff66","Type":"ContainerStarted","Data":"b84c2477541c0b44b06e7dc1d59bef0c858cb1d34b11a20616f997a478e7e52b"} Oct 30 00:38:05.077258 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.076486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:05.077258 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:05.076903 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:05.237235 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.237128 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="f25c62338b9f6e0ceba70a51df57a096bd812cd032d9b49183ff815304b031c4" exitCode=0 Oct 30 00:38:05.237895 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.237236 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerDied","Data":"f25c62338b9f6e0ceba70a51df57a096bd812cd032d9b49183ff815304b031c4"} Oct 30 00:38:05.240442 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.240411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" event={"ID":"08aeeb89-9b10-41a0-a8bf-d07d7642ff66","Type":"ContainerStarted","Data":"19273e951b5729605cc338be1fed10e97d899055e2c84da8c06dc1cbc38dc864"} Oct 30 00:38:05.242260 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.242231 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnlzg" event={"ID":"dd14a22e-0b4d-45eb-a25e-111817cfe34d","Type":"ContainerStarted","Data":"cb0681e2a51f530757e38b1e9dbe6d909d41472ecf8faab4bcad1c17acdba484"} Oct 30 00:38:05.244041 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.244010 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2lt89" event={"ID":"7997439e-9f1e-495d-bac8-6674d4f4a48a","Type":"ContainerStarted","Data":"087d20fbf9045a46e666eab91695d4363bc6f79d84918f134f1da33e81044533"} Oct 30 00:38:05.336010 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.335937 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" podStartSLOduration=2.252403799 podStartE2EDuration="10.335915741s" totalImagesPullingTime="8.083511942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:05.33500962 +0000 UTC m=+11.729649274" watchObservedRunningTime="2026-10-30 00:38:05.335915741 +0000 UTC m=+11.730555397" Oct 30 00:38:05.382044 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:05.381510 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2lt89" podStartSLOduration=2.066447692 podStartE2EDuration="10.381488731s" totalImagesPullingTime="8.315041039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:05.380529418 +0000 UTC m=+11.775169077" watchObservedRunningTime="2026-10-30 00:38:05.381488731 +0000 UTC m=+11.776128386" Oct 30 00:38:06.076936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:06.076885 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:06.076936 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:06.076912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:06.077180 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:06.077042 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:06.077180 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:06.077168 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:06.249717 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:06.249676 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="cb9cfa7cfa949aa5bcdb2299cd1d8c1cb7d268f482d3f29dee9f542c0cef4fd0" exitCode=0 Oct 30 00:38:06.250882 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:06.250859 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerDied","Data":"cb9cfa7cfa949aa5bcdb2299cd1d8c1cb7d268f482d3f29dee9f542c0cef4fd0"} Oct 30 00:38:06.307099 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:06.307034 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qnlzg" podStartSLOduration=2.887006058 podStartE2EDuration="11.307011105s" totalImagesPullingTime="8.420005047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:05.423794571 +0000 UTC m=+11.818434224" watchObservedRunningTime="2026-10-30 00:38:06.307011105 +0000 UTC m=+12.701650762" Oct 30 00:38:07.077396 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:07.077362 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:07.077584 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:07.077508 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:07.253007 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:07.252972 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" event={"ID":"8c300dc5-864d-4faf-ad58-807fb3813895","Type":"ContainerStarted","Data":"c3c0db556233d620c652b76a1a13332a7fad0cb88401b0e63d13193bc084b1fa"} Oct 30 00:38:07.254375 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:07.254352 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4lhrd" event={"ID":"7849b028-2379-4a22-bee9-d1fa3544b3b4","Type":"ContainerStarted","Data":"d35158b4ec9408a78149a3f088ad6208244292aa8205151befe0678af5a70ce6"} Oct 30 00:38:07.295819 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:07.295759 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" podStartSLOduration=1.582801562 podStartE2EDuration="12.295745987s" totalImagesPullingTime="10.712944425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:07.295684966 +0000 UTC m=+13.690324618" watchObservedRunningTime="2026-10-30 00:38:07.295745987 +0000 UTC m=+13.690385636" Oct 30 00:38:07.335552 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:07.335476 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4lhrd" podStartSLOduration=4.109518622 podStartE2EDuration="12.335457255s" totalImagesPullingTime="8.225938633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:07.33480082 +0000 UTC m=+13.729440481" watchObservedRunningTime="2026-10-30 00:38:07.335457255 +0000 UTC m=+13.730096906" Oct 30 00:38:08.076762 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:08.076725 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:08.076943 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:08.076775 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:08.076943 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:08.076872 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:08.077024 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:08.076999 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:08.259996 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:08.259794 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="605b38d68687b51fd1e7903572d1f5fa6e0585a27f3d95ed3661d0ce7be08c4d" exitCode=0 Oct 30 00:38:08.260497 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:08.259886 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerDied","Data":"605b38d68687b51fd1e7903572d1f5fa6e0585a27f3d95ed3661d0ce7be08c4d"} Oct 30 00:38:09.076727 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:09.076690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:09.076934 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:09.076838 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:10.076917 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:10.076779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:10.077420 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:10.076917 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:10.077420 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:10.077288 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:10.077420 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:10.077354 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:10.267580 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:10.267545 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerStarted","Data":"dba62d15d8bf638af0742def77ade22310f38b322ee351dbc998aef456afd2b4"} Oct 30 00:38:11.076337 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:11.076302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:11.076541 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.076425 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:11.272604 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:11.272575 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="dba62d15d8bf638af0742def77ade22310f38b322ee351dbc998aef456afd2b4" exitCode=0 Oct 30 00:38:11.272995 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:11.272617 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerDied","Data":"dba62d15d8bf638af0742def77ade22310f38b322ee351dbc998aef456afd2b4"} Oct 30 00:38:11.849202 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:11.848966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:11.849202 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:11.849191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:11.849394 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.849120 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:38:11.849394 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.849278 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret podName:447c7c04-4097-4065-9e32-ac6b6dc15706 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:27.849260735 +0000 UTC m=+34.243900388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret") pod "global-pull-secret-syncer-vctgv" (UID: "447c7c04-4097-4065-9e32-ac6b6dc15706") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:38:11.849394 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.849280 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:38:11.849394 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.849327 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:27.849312533 +0000 UTC m=+34.243952176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:38:11.950008 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:11.949965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:11.950208 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.950151 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:38:11.950208 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.950177 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:38:11.950208 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.950190 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gplz4 for pod openshift-network-diagnostics/network-check-target-qk7xr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:38:11.950341 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:11.950256 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4 podName:0007937d-ef02-42e5-84de-6d9b8706537d nodeName:}" failed. No retries permitted until 2026-10-30 00:38:27.950238666 +0000 UTC m=+34.344878315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gplz4" (UniqueName: "kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4") pod "network-check-target-qk7xr" (UID: "0007937d-ef02-42e5-84de-6d9b8706537d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:38:12.077018 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.076981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:12.077018 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.077010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:12.077218 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:12.077105 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:12.077249 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:12.077217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:12.277382 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.277291 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"8941a82e1a2820f3f27cda265cd4c82b5abb0b99758f5e17a67a236ef4fac503"} Oct 30 00:38:12.277382 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.277365 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"dd11bcf6deeb8fa0a3625b23e8dbc8ef223256d0454d3f2189e7f9118d95556d"} Oct 30 00:38:12.277958 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.277408 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"3faec123b9af0602cafb1caf3a22b179ace82b3cbf73c6f4ac600e86c1935e2c"} Oct 30 00:38:12.277958 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.277446 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"81448a54db78852e6a92c35db6548bac4c2d7fd2fd53af932e96643d6b7f2ed4"} Oct 30 00:38:12.277958 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.277489 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"5e62bc4f833b46655a2d5fa6c4ee866f36d95a0b80220236a35e9f5ef27fed84"} Oct 30 00:38:12.277958 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.277542 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"1f553c3c8aab54de29c155b6ae276681ca722b5c7fad760a720ca644448f0b2c"} Oct 30 00:38:12.280328 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.280301 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="b0637da489d4f3589f2351d190d68bf0795aa769f67c57ba2f8a733414b3a408" exitCode=0 Oct 30 00:38:12.280432 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:12.280338 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerDied","Data":"b0637da489d4f3589f2351d190d68bf0795aa769f67c57ba2f8a733414b3a408"} Oct 30 00:38:13.076234 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:13.076181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:13.076394 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:13.076308 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:13.286089 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:13.286049 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" event={"ID":"2b542409-f192-467d-92a8-626445065e1b","Type":"ContainerStarted","Data":"a22402e2d1e8ae05b212355873566dcc890b90e2e844048dd9df69d099584ec4"} Oct 30 00:38:13.358407 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:13.358316 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bzs4r" podStartSLOduration=7.821613375 podStartE2EDuration="18.358304009s" totalImagesPullingTime="10.536690634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:13.357804591 +0000 UTC m=+19.752444240" watchObservedRunningTime="2026-10-30 00:38:13.358304009 +0000 UTC m=+19.752943659" Oct 30 00:38:14.077270 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:14.077245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:14.077416 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:14.077336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:14.077467 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:14.077444 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:14.077535 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:14.077336 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:14.291158 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:14.291073 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"31a2597e69f918143e26493bedd6c7933899bf080a950841be52cbfcd2ced1fe"} Oct 30 00:38:15.077133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:15.077100 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:15.077296 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:15.077210 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:16.076810 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:16.076664 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:16.077432 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:16.076690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:16.077432 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:16.076888 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:16.077432 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:16.076956 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:16.299391 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:16.299266 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" event={"ID":"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c","Type":"ContainerStarted","Data":"8a88c65e72201605b7c572b7c8bb487c859ee12248b906e2f102a756cc0fc38f"} Oct 30 00:38:16.299583 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:16.299496 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:16.316024 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:16.315997 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:16.390893 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:16.390836 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" podStartSLOduration=6.359153769 podStartE2EDuration="21.390823943s" totalImagesPullingTime="15.031670174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:16.390449101 +0000 UTC m=+22.785088750" watchObservedRunningTime="2026-10-30 00:38:16.390823943 +0000 UTC m=+22.785463598" Oct 30 00:38:17.076459 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:17.076424 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:17.076646 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:17.076557 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:17.301825 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:17.301798 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:17.302247 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:17.301879 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:17.318016 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:17.317984 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:18.076344 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.076308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:18.076582 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.076324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:18.076582 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:18.076445 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:18.076582 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:18.076477 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:18.156563 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.156501 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vctgv"] Oct 30 00:38:18.161722 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.161682 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk7xr"] Oct 30 00:38:18.162550 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.162508 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwnkz"] Oct 30 00:38:18.162669 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.162649 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:18.162779 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:18.162760 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:18.303331 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.303300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:18.303331 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:18.303339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:18.303832 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:18.303430 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:18.303832 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:18.303484 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:20.076855 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:20.076822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:20.076855 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:20.076844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:20.077262 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:20.076822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:20.077262 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:20.076944 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:20.077262 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:20.076998 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:20.077262 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:20.077077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:22.077226 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:22.077187 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:22.077226 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:22.077211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:22.077869 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:22.077351 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:22.077869 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:22.077395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:22.077869 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:22.077462 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:22.077869 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:22.077536 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:24.077652 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.077465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:24.078082 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.077542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:24.078082 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.077737 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vctgv" podUID="447c7c04-4097-4065-9e32-ac6b6dc15706" Oct 30 00:38:24.078082 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.077576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:24.078082 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.077795 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnkz" podUID="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" Oct 30 00:38:24.078082 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.077860 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk7xr" podUID="0007937d-ef02-42e5-84de-6d9b8706537d" Oct 30 00:38:24.375263 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.375191 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeReady" Oct 30 00:38:24.375394 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.375335 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:38:24.428335 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.428304 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wl87c"] Oct 30 00:38:24.435986 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.435960 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zws9n"] Oct 30 00:38:24.436139 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.436071 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wl87c" Oct 30 00:38:24.438220 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.438199 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zbr9f"] Oct 30 00:38:24.438396 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.438376 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.441418 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.441400 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nzzvk"] Oct 30 00:38:24.441562 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.441545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" Oct 30 00:38:24.445641 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.445614 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg"] Oct 30 00:38:24.445812 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.445704 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" Oct 30 00:38:24.448724 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.448708 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv"] Oct 30 00:38:24.448849 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.448832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.452111 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.452091 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7844cf9dc7-sv5gn"] Oct 30 00:38:24.452195 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.452157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.455014 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.454996 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb"] Oct 30 00:38:24.455209 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.455119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.456709 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.456624 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.456709 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.456652 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.458848 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.458819 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:38:24.458952 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.458896 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.459002 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.458977 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.459700 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.459652 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5"] Oct 30 00:38:24.460498 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.460478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.463346 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.463327 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85b6df665-xt6bh"] Oct 30 00:38:24.463506 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.463442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" Oct 30 00:38:24.468370 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.468350 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp"] Oct 30 00:38:24.468543 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.468507 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.470124 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.470092 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-98rrr\"" Oct 30 00:38:24.471959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.471943 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lzlc8"] Oct 30 00:38:24.472150 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.472126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.478183 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.478154 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:38:24.481179 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.478915 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bdfc8c748-74smf"] Oct 30 00:38:24.481390 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.481325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.482642 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.482622 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vsv6w"] Oct 30 00:38:24.483902 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.483879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.486168 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.486150 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz"] Oct 30 00:38:24.486277 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.486260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:24.491598 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.491579 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xwtsr"] Oct 30 00:38:24.491741 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.491721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" Oct 30 00:38:24.495850 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.494571 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wl87c"] Oct 30 00:38:24.495850 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.494627 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hwvx2"] Oct 30 00:38:24.495850 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.494800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.498892 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.498871 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nzzvk"] Oct 30 00:38:24.499067 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499049 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zws9n"] Oct 30 00:38:24.499144 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499082 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zbr9f"] Oct 30 00:38:24.499144 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499104 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5"] Oct 30 00:38:24.499245 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499176 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv"] Oct 30 00:38:24.499245 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499207 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb"] Oct 30 00:38:24.499245 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499232 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp"] Oct 30 00:38:24.499429 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499253 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lzlc8"] Oct 30 00:38:24.499429 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499295 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b6df665-xt6bh"] Oct 30 00:38:24.499429 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499317 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bdfc8c748-74smf"] Oct 30 00:38:24.499429 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499336 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vsv6w"] Oct 30 00:38:24.499429 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499423 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:24.499701 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.499547 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg"] Oct 30 00:38:24.500338 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.500320 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz"] Oct 30 00:38:24.511454 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.511428 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwvx2"] Oct 30 00:38:24.517330 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.517292 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7844cf9dc7-sv5gn"] Oct 30 00:38:24.527992 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.527968 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xwtsr"] Oct 30 00:38:24.533649 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-config\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.533777 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-tmp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.533777 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmcb\" (UniqueName: \"kubernetes.io/projected/3bc7448f-822d-4574-af12-0545a3513234-kube-api-access-2jmcb\") pod \"volume-data-source-validator-58c9f5f46d-xjbn5\" (UID: \"3bc7448f-822d-4574-af12-0545a3513234\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" Oct 30 00:38:24.533883 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.533883 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.533985 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-default-certificate\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.533985 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.533985 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbftv\" (UniqueName: \"kubernetes.io/projected/2968e47e-4347-42e8-8944-6415c8cef439-kube-api-access-dbftv\") pod \"kas-connection-checker-685895c7c9-wl87c\" (UID: \"2968e47e-4347-42e8-8944-6415c8cef439\") " pod="kube-system/kas-connection-checker-685895c7c9-wl87c" Oct 30 00:38:24.534121 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.533997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.534121 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.534121 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.534261 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.534261 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndvfx\" (UniqueName: \"kubernetes.io/projected/fc3bbd76-9720-4475-8a2f-c8bee23c004b-kube-api-access-ndvfx\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.534261 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c119ce3-3076-4976-8eba-9cc42b321dbc-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.534411 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.534411 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-trusted-ca\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.534411 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-snapshots\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.534411 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppw8v\" (UniqueName: \"kubernetes.io/projected/f8053370-48fa-4a20-8865-96d61d33a0b4-kube-api-access-ppw8v\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.534628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.534628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd0dfbe-2086-4370-8336-051395569bd8-serving-cert\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.534628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.534628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.534628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbfb585c-3327-4d80-b2c3-058e29b0c881-config\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.534628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.534628 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbfb585c-3327-4d80-b2c3-058e29b0c881-serving-cert\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-stats-auth\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clv6k\" (UniqueName: \"kubernetes.io/projected/3c119ce3-3076-4976-8eba-9cc42b321dbc-kube-api-access-clv6k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z99q\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86bb8832-40cc-49f3-b7f2-247e47c55c18-serving-cert\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktxl\" (UniqueName: \"kubernetes.io/projected/fdd0dfbe-2086-4370-8336-051395569bd8-kube-api-access-xktxl\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.534959 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2rp\" (UniqueName: \"kubernetes.io/projected/86bb8832-40cc-49f3-b7f2-247e47c55c18-kube-api-access-qq2rp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.534982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c119ce3-3076-4976-8eba-9cc42b321dbc-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cths\" (UniqueName: \"kubernetes.io/projected/5f621def-706c-43f8-afcc-a79f4441a7c0-kube-api-access-9cths\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8053370-48fa-4a20-8865-96d61d33a0b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-service-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rp99\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48cx\" (UniqueName: \"kubernetes.io/projected/7b1ecc80-c5e0-40ac-b11c-957168289d3b-kube-api-access-h48cx\") pod \"kas-connection-checker-685895c7c9-nzzvk\" (UID: \"7b1ecc80-c5e0-40ac-b11c-957168289d3b\") " pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" Oct 30 00:38:24.535329 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjlr6\" (UniqueName: \"kubernetes.io/projected/cbfb585c-3327-4d80-b2c3-058e29b0c881-kube-api-access-fjlr6\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.535723 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.535723 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.535723 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.535445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfn8\" (UniqueName: \"kubernetes.io/projected/6334ee19-9ae2-42e9-b704-39213b81e260-kube-api-access-wsfn8\") pod \"kas-connection-checker-685895c7c9-zbr9f\" (UID: \"6334ee19-9ae2-42e9-b704-39213b81e260\") " pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" Oct 30 00:38:24.557324 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.557298 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.557768 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.557741 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.557856 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.557799 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 00:38:24.557856 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.557814 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.557960 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.557864 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.557960 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.557879 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:38:24.558018 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.557978 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 00:38:24.558303 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558283 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.558303 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558297 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.558303 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558300 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558345 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558351 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558353 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558407 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558407 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558421 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558451 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558460 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558497 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.558499 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558501 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 00:38:24.558914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558553 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.558914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558555 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.558914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558629 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.558914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558748 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.558914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.558781 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 00:38:24.562106 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.560797 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 00:38:24.562106 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.561918 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 00:38:24.562421 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.562400 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 00:38:24.563183 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.563164 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 00:38:24.563900 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.563824 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 00:38:24.564570 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.564552 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:38:24.564947 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.564925 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 00:38:24.565544 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.565508 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 00:38:24.570963 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.570947 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 00:38:24.571125 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571104 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 00:38:24.571125 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571106 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 00:38:24.571295 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571140 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 00:38:24.571295 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571146 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9cwfv\"" Oct 30 00:38:24.571295 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571256 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571304 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h424n\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571313 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tswvk\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571320 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-gsl54\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571346 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4hm7n\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571361 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571375 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tb2kx\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571379 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571397 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7gr5j\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571428 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jxbz5\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571438 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8h57b\"" Oct 30 00:38:24.571448 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571430 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mxnln\"" Oct 30 00:38:24.571859 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571625 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qbmx5\"" Oct 30 00:38:24.571859 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.571826 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-drt9j\"" Oct 30 00:38:24.572606 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.572430 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 00:38:24.572606 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.572435 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 00:38:24.573505 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.573484 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 00:38:24.636842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.636763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rp99\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.636842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.636802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48cx\" (UniqueName: \"kubernetes.io/projected/7b1ecc80-c5e0-40ac-b11c-957168289d3b-kube-api-access-h48cx\") pod \"kas-connection-checker-685895c7c9-nzzvk\" (UID: \"7b1ecc80-c5e0-40ac-b11c-957168289d3b\") " pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" Oct 30 00:38:24.636842 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.636820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlr6\" (UniqueName: \"kubernetes.io/projected/cbfb585c-3327-4d80-b2c3-058e29b0c881-kube-api-access-fjlr6\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.637133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.636851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.637133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.636871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.637133 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.636957 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:24.637133 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.637027 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.137009008 +0000 UTC m=+31.531648656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:24.637133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfn8\" (UniqueName: \"kubernetes.io/projected/6334ee19-9ae2-42e9-b704-39213b81e260-kube-api-access-wsfn8\") pod \"kas-connection-checker-685895c7c9-zbr9f\" (UID: \"6334ee19-9ae2-42e9-b704-39213b81e260\") " pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" Oct 30 00:38:24.637133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-config\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.637133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637129 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70d2e011-58cb-4646-adb9-04f8eacab25c-config-volume\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.637434 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wjh5\" (UniqueName: \"kubernetes.io/projected/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-kube-api-access-6wjh5\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:24.637434 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-tmp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.637434 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmcb\" (UniqueName: \"kubernetes.io/projected/3bc7448f-822d-4574-af12-0545a3513234-kube-api-access-2jmcb\") pod \"volume-data-source-validator-58c9f5f46d-xjbn5\" (UID: \"3bc7448f-822d-4574-af12-0545a3513234\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" Oct 30 00:38:24.637434 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.637434 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.637434 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-default-certificate\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.637469 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.137449811 +0000 UTC m=+31.532089460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbftv\" (UniqueName: \"kubernetes.io/projected/2968e47e-4347-42e8-8944-6415c8cef439-kube-api-access-dbftv\") pod \"kas-connection-checker-685895c7c9-wl87c\" (UID: \"2968e47e-4347-42e8-8944-6415c8cef439\") " pod="kube-system/kas-connection-checker-685895c7c9-wl87c" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637620 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-tmp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndvfx\" (UniqueName: \"kubernetes.io/projected/fc3bbd76-9720-4475-8a2f-c8bee23c004b-kube-api-access-ndvfx\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.637732 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c119ce3-3076-4976-8eba-9cc42b321dbc-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-trusted-ca\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-snapshots\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637821 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppw8v\" (UniqueName: \"kubernetes.io/projected/f8053370-48fa-4a20-8865-96d61d33a0b4-kube-api-access-ppw8v\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd0dfbe-2086-4370-8336-051395569bd8-serving-cert\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637950 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-config\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.638028 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.638041 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.637951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.638082 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.138068194 +0000 UTC m=+31.532707867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbfb585c-3327-4d80-b2c3-058e29b0c881-config\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.638232 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbfb585c-3327-4d80-b2c3-058e29b0c881-serving-cert\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638175 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-stats-auth\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/abb2b77c-ea98-49f5-a77d-f531c74980c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clv6k\" (UniqueName: \"kubernetes.io/projected/3c119ce3-3076-4976-8eba-9cc42b321dbc-kube-api-access-clv6k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86h2w\" (UniqueName: \"kubernetes.io/projected/936fadee-4c5d-4ca7-b4b8-9d107c949645-kube-api-access-86h2w\") pod \"network-check-source-6f78d59f99-gr6hz\" (UID: \"936fadee-4c5d-4ca7-b4b8-9d107c949645\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z99q\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d2e011-58cb-4646-adb9-04f8eacab25c-tmp-dir\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnv77\" (UniqueName: \"kubernetes.io/projected/70d2e011-58cb-4646-adb9-04f8eacab25c-kube-api-access-gnv77\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.639012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-cert\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86bb8832-40cc-49f3-b7f2-247e47c55c18-serving-cert\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktxl\" (UniqueName: \"kubernetes.io/projected/fdd0dfbe-2086-4370-8336-051395569bd8-kube-api-access-xktxl\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2rp\" (UniqueName: \"kubernetes.io/projected/86bb8832-40cc-49f3-b7f2-247e47c55c18-kube-api-access-qq2rp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c119ce3-3076-4976-8eba-9cc42b321dbc-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cths\" (UniqueName: \"kubernetes.io/projected/5f621def-706c-43f8-afcc-a79f4441a7c0-kube-api-access-9cths\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8053370-48fa-4a20-8865-96d61d33a0b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-service-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.638852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.639070 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-trusted-ca\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.639101 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.639163 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:24.639737 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.639216 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.139201388 +0000 UTC m=+31.533841026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:24.640482 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.639213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.640482 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.639313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c119ce3-3076-4976-8eba-9cc42b321dbc-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.640482 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.639508 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-snapshots\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.640482 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.639631 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:24.640482 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.639685 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.139671446 +0000 UTC m=+31.534311078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:24.640482 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.639768 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbfb585c-3327-4d80-b2c3-058e29b0c881-config\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.642349 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.641971 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:24.642349 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.641989 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:24.642349 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.641983 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.642349 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.642042 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.1420275 +0000 UTC m=+31.536667145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:24.642767 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.642435 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-service-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.642767 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.642619 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8053370-48fa-4a20-8865-96d61d33a0b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.643022 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.642964 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbfb585c-3327-4d80-b2c3-058e29b0c881-serving-cert\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.643022 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.642966 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd0dfbe-2086-4370-8336-051395569bd8-serving-cert\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.643336 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.643314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.643410 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.643390 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.643856 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.643830 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.643972 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.643956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86bb8832-40cc-49f3-b7f2-247e47c55c18-serving-cert\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.644562 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.644510 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.644922 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.644904 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-stats-auth\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.645238 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.645214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c119ce3-3076-4976-8eba-9cc42b321dbc-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.645496 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.645478 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.645604 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.645589 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-default-certificate\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.661008 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.660140 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48cx\" (UniqueName: \"kubernetes.io/projected/7b1ecc80-c5e0-40ac-b11c-957168289d3b-kube-api-access-h48cx\") pod \"kas-connection-checker-685895c7c9-nzzvk\" (UID: \"7b1ecc80-c5e0-40ac-b11c-957168289d3b\") " pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" Oct 30 00:38:24.661008 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.660566 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlr6\" (UniqueName: \"kubernetes.io/projected/cbfb585c-3327-4d80-b2c3-058e29b0c881-kube-api-access-fjlr6\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.661255 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.661148 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rp99\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.661430 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.661401 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbftv\" (UniqueName: \"kubernetes.io/projected/2968e47e-4347-42e8-8944-6415c8cef439-kube-api-access-dbftv\") pod \"kas-connection-checker-685895c7c9-wl87c\" (UID: \"2968e47e-4347-42e8-8944-6415c8cef439\") " pod="kube-system/kas-connection-checker-685895c7c9-wl87c" Oct 30 00:38:24.662550 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.662487 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.662644 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.662602 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktxl\" (UniqueName: \"kubernetes.io/projected/fdd0dfbe-2086-4370-8336-051395569bd8-kube-api-access-xktxl\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.663493 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.663463 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:24.663904 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.663468 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clv6k\" (UniqueName: \"kubernetes.io/projected/3c119ce3-3076-4976-8eba-9cc42b321dbc-kube-api-access-clv6k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.663977 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.663902 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfn8\" (UniqueName: \"kubernetes.io/projected/6334ee19-9ae2-42e9-b704-39213b81e260-kube-api-access-wsfn8\") pod \"kas-connection-checker-685895c7c9-zbr9f\" (UID: \"6334ee19-9ae2-42e9-b704-39213b81e260\") " pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" Oct 30 00:38:24.664048 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.664026 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndvfx\" (UniqueName: \"kubernetes.io/projected/fc3bbd76-9720-4475-8a2f-c8bee23c004b-kube-api-access-ndvfx\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:24.664100 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.664074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppw8v\" (UniqueName: \"kubernetes.io/projected/f8053370-48fa-4a20-8865-96d61d33a0b4-kube-api-access-ppw8v\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:24.664428 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.664404 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cths\" (UniqueName: \"kubernetes.io/projected/5f621def-706c-43f8-afcc-a79f4441a7c0-kube-api-access-9cths\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:24.665414 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.665395 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2rp\" (UniqueName: \"kubernetes.io/projected/86bb8832-40cc-49f3-b7f2-247e47c55c18-kube-api-access-qq2rp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.665652 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.665634 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z99q\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:24.666386 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.666369 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmcb\" (UniqueName: \"kubernetes.io/projected/3bc7448f-822d-4574-af12-0545a3513234-kube-api-access-2jmcb\") pod \"volume-data-source-validator-58c9f5f46d-xjbn5\" (UID: \"3bc7448f-822d-4574-af12-0545a3513234\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" Oct 30 00:38:24.740051 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70d2e011-58cb-4646-adb9-04f8eacab25c-config-volume\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.740051 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjh5\" (UniqueName: \"kubernetes.io/projected/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-kube-api-access-6wjh5\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:24.740431 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/abb2b77c-ea98-49f5-a77d-f531c74980c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:24.740562 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:24.740562 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.740562 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86h2w\" (UniqueName: \"kubernetes.io/projected/936fadee-4c5d-4ca7-b4b8-9d107c949645-kube-api-access-86h2w\") pod \"network-check-source-6f78d59f99-gr6hz\" (UID: \"936fadee-4c5d-4ca7-b4b8-9d107c949645\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d2e011-58cb-4646-adb9-04f8eacab25c-tmp-dir\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.740578 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnv77\" (UniqueName: \"kubernetes.io/projected/70d2e011-58cb-4646-adb9-04f8eacab25c-kube-api-access-gnv77\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.740645 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert podName:abb2b77c-ea98-49f5-a77d-f531c74980c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.240630094 +0000 UTC m=+31.635269736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vsv6w" (UID: "abb2b77c-ea98-49f5-a77d-f531c74980c0") : secret "networking-console-plugin-cert" not found Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-cert\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.740671 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.740677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70d2e011-58cb-4646-adb9-04f8eacab25c-config-volume\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.740729 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:24.740719 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls podName:70d2e011-58cb-4646-adb9-04f8eacab25c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:25.240703973 +0000 UTC m=+31.635343621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls") pod "dns-default-xwtsr" (UID: "70d2e011-58cb-4646-adb9-04f8eacab25c") : secret "dns-default-metrics-tls" not found Oct 30 00:38:24.742751 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.742732 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d2e011-58cb-4646-adb9-04f8eacab25c-tmp-dir\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.743144 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.743123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/abb2b77c-ea98-49f5-a77d-f531c74980c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:24.744858 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.744843 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-cert\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:24.750437 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.750416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wl87c" Oct 30 00:38:24.756130 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.756111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:24.761632 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.761607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjh5\" (UniqueName: \"kubernetes.io/projected/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-kube-api-access-6wjh5\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:24.761723 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.761646 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnv77\" (UniqueName: \"kubernetes.io/projected/70d2e011-58cb-4646-adb9-04f8eacab25c-kube-api-access-gnv77\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:24.761898 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.761879 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86h2w\" (UniqueName: \"kubernetes.io/projected/936fadee-4c5d-4ca7-b4b8-9d107c949645-kube-api-access-86h2w\") pod \"network-check-source-6f78d59f99-gr6hz\" (UID: \"936fadee-4c5d-4ca7-b4b8-9d107c949645\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" Oct 30 00:38:24.765075 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.765057 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" Oct 30 00:38:24.777258 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.777234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" Oct 30 00:38:24.817572 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.817512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:24.841319 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.841291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" Oct 30 00:38:24.899379 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.897919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:24.919701 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.915089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:24.938694 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.938667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" Oct 30 00:38:24.982733 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:24.982699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:25.144846 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.144767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:25.144846 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.144817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:25.144846 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.144844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.144877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.144895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.144922 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.144982 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.144966841 +0000 UTC m=+32.539606489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.144967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.144996 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145016 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.144996368 +0000 UTC m=+32.539636016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145001 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145038 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145049 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145048 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145048 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.145035105 +0000 UTC m=+32.539674748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145080 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145094 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.145082767 +0000 UTC m=+32.539722404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145107 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.145101191 +0000 UTC m=+32.539740820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:25.145328 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.145119 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.145113861 +0000 UTC m=+32.539753490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:25.152375 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.152349 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zws9n"] Oct 30 00:38:25.154041 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:25.154019 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bb8832_40cc_49f3_b7f2_247e47c55c18.slice/crio-f47a4c498bb558652049ef528b129b2e568f9e99518b2411763b2ca210685063 WatchSource:0}: Error finding container f47a4c498bb558652049ef528b129b2e568f9e99518b2411763b2ca210685063: Status 404 returned error can't find the container with id f47a4c498bb558652049ef528b129b2e568f9e99518b2411763b2ca210685063 Oct 30 00:38:25.157902 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.157881 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nzzvk"] Oct 30 00:38:25.159028 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:25.159005 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1ecc80_c5e0_40ac_b11c_957168289d3b.slice/crio-eaea34fcbcaefa793c7b8bb4c29610f252afe5f0f7f8db55582825ba0fb2e97a WatchSource:0}: Error finding container eaea34fcbcaefa793c7b8bb4c29610f252afe5f0f7f8db55582825ba0fb2e97a: Status 404 returned error can't find the container with id eaea34fcbcaefa793c7b8bb4c29610f252afe5f0f7f8db55582825ba0fb2e97a Oct 30 00:38:25.163263 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.163241 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wl87c"] Oct 30 00:38:25.163966 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:25.163946 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2968e47e_4347_42e8_8944_6415c8cef439.slice/crio-51b4e1462bc7c4f63fa303067b49e27d6ff548dad1a1bfb58fe285b96823a98f WatchSource:0}: Error finding container 51b4e1462bc7c4f63fa303067b49e27d6ff548dad1a1bfb58fe285b96823a98f: Status 404 returned error can't find the container with id 51b4e1462bc7c4f63fa303067b49e27d6ff548dad1a1bfb58fe285b96823a98f Oct 30 00:38:25.194377 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.194142 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zbr9f"] Oct 30 00:38:25.199335 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.199308 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb"] Oct 30 00:38:25.245930 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.245896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:25.246097 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.245940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:25.246186 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.246170 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:38:25.246267 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.246236 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls podName:70d2e011-58cb-4646-adb9-04f8eacab25c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.246219284 +0000 UTC m=+32.640858933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls") pod "dns-default-xwtsr" (UID: "70d2e011-58cb-4646-adb9-04f8eacab25c") : secret "dns-default-metrics-tls" not found Oct 30 00:38:25.246450 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.246426 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:38:25.246509 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:25.246479 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert podName:abb2b77c-ea98-49f5-a77d-f531c74980c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:26.246465157 +0000 UTC m=+32.641104803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vsv6w" (UID: "abb2b77c-ea98-49f5-a77d-f531c74980c0") : secret "networking-console-plugin-cert" not found Oct 30 00:38:25.252148 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.252123 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5"] Oct 30 00:38:25.258876 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:25.258851 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc7448f_822d_4574_af12_0545a3513234.slice/crio-a9079f58e0915c79c36d52a1162f76f7be696333eb3ae20fd0acff33ab10e218 WatchSource:0}: Error finding container a9079f58e0915c79c36d52a1162f76f7be696333eb3ae20fd0acff33ab10e218: Status 404 returned error can't find the container with id a9079f58e0915c79c36d52a1162f76f7be696333eb3ae20fd0acff33ab10e218 Oct 30 00:38:25.271337 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.271309 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp"] Oct 30 00:38:25.284648 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.283849 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz"] Oct 30 00:38:25.284648 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.283910 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lzlc8"] Oct 30 00:38:25.313822 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.313741 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwvx2"] Oct 30 00:38:25.322831 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.322739 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wl87c" event={"ID":"2968e47e-4347-42e8-8944-6415c8cef439","Type":"ContainerStarted","Data":"51b4e1462bc7c4f63fa303067b49e27d6ff548dad1a1bfb58fe285b96823a98f"} Oct 30 00:38:25.324348 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.324321 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" event={"ID":"cbfb585c-3327-4d80-b2c3-058e29b0c881","Type":"ContainerStarted","Data":"a24e5e110cb085dcec6f33bb38ae7ced3d5e531b0660f605be2656810e0dffe4"} Oct 30 00:38:25.327749 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.327722 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" event={"ID":"fdd0dfbe-2086-4370-8336-051395569bd8","Type":"ContainerStarted","Data":"e08f16ff0bbc0ef7cdb346c41c236170072938f63ddf8419d9cce2339720c348"} Oct 30 00:38:25.329750 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.329720 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" event={"ID":"6334ee19-9ae2-42e9-b704-39213b81e260","Type":"ContainerStarted","Data":"e0d538f05b7b05060cac4bb736cc111723c8aea3f71b5b9940e5b650c556a6f5"} Oct 30 00:38:25.331638 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.331611 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zws9n" event={"ID":"86bb8832-40cc-49f3-b7f2-247e47c55c18","Type":"ContainerStarted","Data":"f47a4c498bb558652049ef528b129b2e568f9e99518b2411763b2ca210685063"} Oct 30 00:38:25.332919 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.332859 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" event={"ID":"3bc7448f-822d-4574-af12-0545a3513234","Type":"ContainerStarted","Data":"a9079f58e0915c79c36d52a1162f76f7be696333eb3ae20fd0acff33ab10e218"} Oct 30 00:38:25.334762 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.334739 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" event={"ID":"7b1ecc80-c5e0-40ac-b11c-957168289d3b","Type":"ContainerStarted","Data":"eaea34fcbcaefa793c7b8bb4c29610f252afe5f0f7f8db55582825ba0fb2e97a"} Oct 30 00:38:25.336243 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.336184 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" event={"ID":"936fadee-4c5d-4ca7-b4b8-9d107c949645","Type":"ContainerStarted","Data":"5b6808f532f675366d24340b0359d1fb63229390502c572174b1801b42ee02d3"} Oct 30 00:38:25.337482 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:25.337461 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" event={"ID":"3c119ce3-3076-4976-8eba-9cc42b321dbc","Type":"ContainerStarted","Data":"c53f95c650962bcd2bdee87475747a1f40e183257be07217f16332b25201c23d"} Oct 30 00:38:25.339538 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:25.339495 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee817c_3a6d_46d6_8ae3_d0fd7417e978.slice/crio-b6076c8eb82be2cf9bb3759a9239e64c5019cdd36d7379871e2d80838139f857 WatchSource:0}: Error finding container b6076c8eb82be2cf9bb3759a9239e64c5019cdd36d7379871e2d80838139f857: Status 404 returned error can't find the container with id b6076c8eb82be2cf9bb3759a9239e64c5019cdd36d7379871e2d80838139f857 Oct 30 00:38:26.079264 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.079226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:26.079752 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.079730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:26.080141 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.080119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:26.096505 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.096235 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-smcj2\"" Oct 30 00:38:26.096505 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.096384 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:38:26.096505 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.096436 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-44cn9\"" Oct 30 00:38:26.096505 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.096452 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:38:26.096505 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.096481 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9xms5\"" Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.156060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.156167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.156226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.156280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.156316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.156370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.156504 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.156542 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.156600 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.15658134 +0000 UTC m=+34.551220974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157042 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157101 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.157080432 +0000 UTC m=+34.551720066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157166 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157176 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157237 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.157223209 +0000 UTC m=+34.551862842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157294 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:26.157631 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157326 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.157316269 +0000 UTC m=+34.551955902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:26.158899 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157479 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.157469174 +0000 UTC m=+34.552108809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:26.158899 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157557 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:26.158899 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.157591 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.157578735 +0000 UTC m=+34.552218370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:26.258887 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.257993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:26.258887 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.258043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:26.258887 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.258677 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:38:26.258887 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.258753 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert podName:abb2b77c-ea98-49f5-a77d-f531c74980c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.258733171 +0000 UTC m=+34.653372805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vsv6w" (UID: "abb2b77c-ea98-49f5-a77d-f531c74980c0") : secret "networking-console-plugin-cert" not found Oct 30 00:38:26.258887 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.258820 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:38:26.258887 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:26.258853 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls podName:70d2e011-58cb-4646-adb9-04f8eacab25c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:28.258842216 +0000 UTC m=+34.653481855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls") pod "dns-default-xwtsr" (UID: "70d2e011-58cb-4646-adb9-04f8eacab25c") : secret "dns-default-metrics-tls" not found Oct 30 00:38:26.349969 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.349842 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wl87c" event={"ID":"2968e47e-4347-42e8-8944-6415c8cef439","Type":"ContainerStarted","Data":"6272e0d1bad575975803dc8b3f3aa327412353eadd72b4601f82685a4507e8e7"} Oct 30 00:38:26.355615 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.354993 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" event={"ID":"6334ee19-9ae2-42e9-b704-39213b81e260","Type":"ContainerStarted","Data":"d8b758e6f8289542ee1e89312ebacf89fadb325cbbbb71422296cea94e80240d"} Oct 30 00:38:26.358140 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.358020 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwvx2" event={"ID":"aaee817c-3a6d-46d6-8ae3-d0fd7417e978","Type":"ContainerStarted","Data":"b6076c8eb82be2cf9bb3759a9239e64c5019cdd36d7379871e2d80838139f857"} Oct 30 00:38:26.364885 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.364351 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" event={"ID":"7b1ecc80-c5e0-40ac-b11c-957168289d3b","Type":"ContainerStarted","Data":"85de2f218c5df61516f4bdba0eddb3c33d5b42364bc91e2d798ba57444e9f3b5"} Oct 30 00:38:26.442243 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.442160 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wl87c" podStartSLOduration=344.442136945 podStartE2EDuration="5m44.442136945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:26.400757753 +0000 UTC m=+32.795397403" watchObservedRunningTime="2026-10-30 00:38:26.442136945 +0000 UTC m=+32.836776592" Oct 30 00:38:26.484788 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.484723 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" podStartSLOduration=344.4847019 podStartE2EDuration="5m44.4847019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:26.483751784 +0000 UTC m=+32.878391439" watchObservedRunningTime="2026-10-30 00:38:26.4847019 +0000 UTC m=+32.879341552" Oct 30 00:38:26.486012 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:26.485279 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" podStartSLOduration=344.485266781 podStartE2EDuration="5m44.485266781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:26.443084119 +0000 UTC m=+32.837723771" watchObservedRunningTime="2026-10-30 00:38:26.485266781 +0000 UTC m=+32.879906429" Oct 30 00:38:27.877267 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:27.877187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:27.877824 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:27.877272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:27.877824 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:27.877352 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:38:27.877824 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:27.877423 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:59.877405884 +0000 UTC m=+66.272045514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : secret "metrics-daemon-secret" not found Oct 30 00:38:27.881537 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:27.881484 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:27.950751 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:27.950714 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:27.979142 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:27.978537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:27.983478 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:27.983351 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.180409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.180481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.180555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.180596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.180639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.180674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.180853 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.180921 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.180903397 +0000 UTC m=+38.575543050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181343 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181359 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181407 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.181392504 +0000 UTC m=+38.576032156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181474 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181513 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.181500324 +0000 UTC m=+38.576139966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181599 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.18158858 +0000 UTC m=+38.576228230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:28.181879 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181658 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:28.182791 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.181681843 +0000 UTC m=+38.576321486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:28.182791 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181749 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:28.182791 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181761 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:28.182791 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.181789 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.181779163 +0000 UTC m=+38.576418807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:28.266385 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.266348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:28.281782 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.281751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:28.281950 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.281790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:28.282063 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.282046 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:38:28.282130 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.282108 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls podName:70d2e011-58cb-4646-adb9-04f8eacab25c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.282091158 +0000 UTC m=+38.676730801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls") pod "dns-default-xwtsr" (UID: "70d2e011-58cb-4646-adb9-04f8eacab25c") : secret "dns-default-metrics-tls" not found Oct 30 00:38:28.282195 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.282175 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:38:28.282244 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:28.282211 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert podName:abb2b77c-ea98-49f5-a77d-f531c74980c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.2821998 +0000 UTC m=+38.676839443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vsv6w" (UID: "abb2b77c-ea98-49f5-a77d-f531c74980c0") : secret "networking-console-plugin-cert" not found Oct 30 00:38:28.297154 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.296958 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vctgv"] Oct 30 00:38:28.307396 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:28.307345 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447c7c04_4097_4065_9e32_ac6b6dc15706.slice/crio-720bf01b0fd486f50696f0f12652964caafc24584b8101a10485df447ffd8a01 WatchSource:0}: Error finding container 720bf01b0fd486f50696f0f12652964caafc24584b8101a10485df447ffd8a01: Status 404 returned error can't find the container with id 720bf01b0fd486f50696f0f12652964caafc24584b8101a10485df447ffd8a01 Oct 30 00:38:28.385178 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.384383 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" event={"ID":"3bc7448f-822d-4574-af12-0545a3513234","Type":"ContainerStarted","Data":"c2c0d7c0faa74450b1eb1c9aa1e1f4db5653a7ef2d27b7b3374d429ff23337d4"} Oct 30 00:38:28.398344 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.393621 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vctgv" event={"ID":"447c7c04-4097-4065-9e32-ac6b6dc15706","Type":"ContainerStarted","Data":"720bf01b0fd486f50696f0f12652964caafc24584b8101a10485df447ffd8a01"} Oct 30 00:38:28.608094 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.607449 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" podStartSLOduration=319.337863616 podStartE2EDuration="5m21.607425865s" totalImagesPullingTime="2.269562249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:28.427215463 +0000 UTC m=+34.821855114" watchObservedRunningTime="2026-10-30 00:38:28.607425865 +0000 UTC m=+35.002065517" Oct 30 00:38:28.608094 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:28.607993 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk7xr"] Oct 30 00:38:28.610347 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:28.610306 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0007937d_ef02_42e5_84de_6d9b8706537d.slice/crio-76516f609addbf05d522d0cd242ca9d4e0a9bcaa17113e260f09ebc9ba6e115c WatchSource:0}: Error finding container 76516f609addbf05d522d0cd242ca9d4e0a9bcaa17113e260f09ebc9ba6e115c: Status 404 returned error can't find the container with id 76516f609addbf05d522d0cd242ca9d4e0a9bcaa17113e260f09ebc9ba6e115c Oct 30 00:38:29.402914 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:29.402838 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zws9n" event={"ID":"86bb8832-40cc-49f3-b7f2-247e47c55c18","Type":"ContainerStarted","Data":"a44b5c74fb28bf50d6bcd45af47f497b24d72e6cfebeb26a93ec0caac4689cc5"} Oct 30 00:38:29.406113 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:29.406076 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk7xr" event={"ID":"0007937d-ef02-42e5-84de-6d9b8706537d","Type":"ContainerStarted","Data":"76516f609addbf05d522d0cd242ca9d4e0a9bcaa17113e260f09ebc9ba6e115c"} Oct 30 00:38:29.451009 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:29.450944 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zws9n" podStartSLOduration=318.480435074 podStartE2EDuration="5m22.450928693s" totalImagesPullingTime="3.970493619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:29.4507012 +0000 UTC m=+35.845340851" watchObservedRunningTime="2026-10-30 00:38:29.450928693 +0000 UTC m=+35.845568344" Oct 30 00:38:30.413895 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.413810 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" event={"ID":"3c119ce3-3076-4976-8eba-9cc42b321dbc","Type":"ContainerStarted","Data":"c1ea50be29fa2ff0898b95ed1aae929c26a917baf44ad2cbf7b25a2b5a4995d7"} Oct 30 00:38:30.420182 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.420143 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" event={"ID":"cbfb585c-3327-4d80-b2c3-058e29b0c881","Type":"ContainerStarted","Data":"b6741522c805278b29fd5009314cff26ebff1b6ee965d54a2d6ff50c418590ee"} Oct 30 00:38:30.425927 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.425883 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/0.log" Oct 30 00:38:30.425927 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.425929 2577 generic.go:356] "Generic (PLEG): container finished" podID="fdd0dfbe-2086-4370-8336-051395569bd8" containerID="d54b41a020ed12485808e5e71b825b31d05fdc26a5ec200748352a7c818ab0f4" exitCode=255 Oct 30 00:38:30.426165 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.426129 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" event={"ID":"fdd0dfbe-2086-4370-8336-051395569bd8","Type":"ContainerDied","Data":"d54b41a020ed12485808e5e71b825b31d05fdc26a5ec200748352a7c818ab0f4"} Oct 30 00:38:30.426789 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.426768 2577 scope.go:122] "RemoveContainer" containerID="d54b41a020ed12485808e5e71b825b31d05fdc26a5ec200748352a7c818ab0f4" Oct 30 00:38:30.457276 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.457213 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" podStartSLOduration=318.953858853 podStartE2EDuration="5m23.457197092s" totalImagesPullingTime="4.503338239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:30.456565193 +0000 UTC m=+36.851204845" watchObservedRunningTime="2026-10-30 00:38:30.457197092 +0000 UTC m=+36.851836758" Oct 30 00:38:30.505145 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.505081 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" podStartSLOduration=318.672810338 podStartE2EDuration="5m23.505061057s" totalImagesPullingTime="4.832250719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:30.50457566 +0000 UTC m=+36.899215336" watchObservedRunningTime="2026-10-30 00:38:30.505061057 +0000 UTC m=+36.899700709" Oct 30 00:38:30.566546 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.565646 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhms"] Oct 30 00:38:30.586289 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.586259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.602415 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.602385 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhms"] Oct 30 00:38:30.613510 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.613454 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:38:30.625358 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.625329 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:38:30.625612 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.625580 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqzrd\"" Oct 30 00:38:30.709966 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.709931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b55144-3ccd-406b-a9d2-6be38a04586e-crio-socket\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.710101 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.710070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b55144-3ccd-406b-a9d2-6be38a04586e-data-volume\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.710142 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.710103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.710189 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.710171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.710230 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.710201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsg4l\" (UniqueName: \"kubernetes.io/projected/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-api-access-xsg4l\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.811734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b55144-3ccd-406b-a9d2-6be38a04586e-data-volume\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.811783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.811825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.811839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsg4l\" (UniqueName: \"kubernetes.io/projected/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-api-access-xsg4l\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.811900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b55144-3ccd-406b-a9d2-6be38a04586e-crio-socket\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.812011 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b55144-3ccd-406b-a9d2-6be38a04586e-crio-socket\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.812247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b55144-3ccd-406b-a9d2-6be38a04586e-data-volume\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:30.812365 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:30.812441 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:31.31242142 +0000 UTC m=+37.707061064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:30.812507 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:30.812599 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:30.812564 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:31.312552953 +0000 UTC m=+37.707192587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:30.848133 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.848065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsg4l\" (UniqueName: \"kubernetes.io/projected/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-api-access-xsg4l\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:30.873498 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.873460 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2"] Oct 30 00:38:30.891207 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.891128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" Oct 30 00:38:30.896763 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.896732 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2"] Oct 30 00:38:30.998878 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.998849 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:38:30.998878 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.998849 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:30.999125 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:30.998955 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2jw8t\"" Oct 30 00:38:31.014113 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.013893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9fq\" (UniqueName: \"kubernetes.io/projected/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5-kube-api-access-rj9fq\") pod \"migrator-66df5d496b-skjm2\" (UID: \"d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" Oct 30 00:38:31.115073 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.115031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9fq\" (UniqueName: \"kubernetes.io/projected/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5-kube-api-access-rj9fq\") pod \"migrator-66df5d496b-skjm2\" (UID: \"d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" Oct 30 00:38:31.138267 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.138229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9fq\" (UniqueName: \"kubernetes.io/projected/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5-kube-api-access-rj9fq\") pod \"migrator-66df5d496b-skjm2\" (UID: \"d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" Oct 30 00:38:31.250723 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.250414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" Oct 30 00:38:31.317330 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.317292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:31.317536 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.317351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:31.317596 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:31.317512 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:31.317640 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:31.317605 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.317584214 +0000 UTC m=+38.712223926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:31.317960 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:31.317939 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:31.318073 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:31.317996 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.317981015 +0000 UTC m=+38.712620648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:31.432022 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.431942 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/1.log" Oct 30 00:38:31.432488 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.432431 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/0.log" Oct 30 00:38:31.432488 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.432463 2577 generic.go:356] "Generic (PLEG): container finished" podID="fdd0dfbe-2086-4370-8336-051395569bd8" containerID="8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd" exitCode=255 Oct 30 00:38:31.432653 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.432559 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" event={"ID":"fdd0dfbe-2086-4370-8336-051395569bd8","Type":"ContainerDied","Data":"8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd"} Oct 30 00:38:31.432653 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.432630 2577 scope.go:122] "RemoveContainer" containerID="d54b41a020ed12485808e5e71b825b31d05fdc26a5ec200748352a7c818ab0f4" Oct 30 00:38:31.433040 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.433011 2577 scope.go:122] "RemoveContainer" containerID="8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd" Oct 30 00:38:31.433242 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:31.433220 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lzlc8_openshift-console-operator(fdd0dfbe-2086-4370-8336-051395569bd8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" podUID="fdd0dfbe-2086-4370-8336-051395569bd8" Oct 30 00:38:31.436214 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.436189 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwvx2" event={"ID":"aaee817c-3a6d-46d6-8ae3-d0fd7417e978","Type":"ContainerStarted","Data":"d47f9ee5168c5a625f445eb1d98dfc1baa97cd994277aa9d288a6db0e17e75b7"} Oct 30 00:38:31.518487 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.518383 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hwvx2" podStartSLOduration=2.193558097 podStartE2EDuration="7.518363209s" totalImagesPullingTime="5.324805112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:31.516357452 +0000 UTC m=+37.910997106" watchObservedRunningTime="2026-10-30 00:38:31.518363209 +0000 UTC m=+37.913002860" Oct 30 00:38:31.583657 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:31.583620 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2"] Oct 30 00:38:31.591696 ip-10-0-130-171 kubenswrapper[2577]: W1030 00:38:31.591666 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03b11d9_a4a1_496a_b6ea_6bc8a11bbea5.slice/crio-fc89fc096922a4da0bcab717fa1fde6556060f2268d5d959ef7a8b26d5b88820 WatchSource:0}: Error finding container fc89fc096922a4da0bcab717fa1fde6556060f2268d5d959ef7a8b26d5b88820: Status 404 returned error can't find the container with id fc89fc096922a4da0bcab717fa1fde6556060f2268d5d959ef7a8b26d5b88820 Oct 30 00:38:32.225800 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.225775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:32.225908 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.225838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:32.225908 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.225887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:32.226007 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.225910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:32.226007 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.225916 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:32.226007 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.225933 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:32.226007 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.225950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:32.226007 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.225976 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.225962126 +0000 UTC m=+46.620601755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:32.226007 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.225987 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:32.226007 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.225995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226024 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226029 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.226018669 +0000 UTC m=+46.620658299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226057 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226065 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226078 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226090 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.226083992 +0000 UTC m=+46.620723621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226102 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.226097121 +0000 UTC m=+46.620736750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226122 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.226108196 +0000 UTC m=+46.620747839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:32.226265 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.226142 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.226132976 +0000 UTC m=+46.620772605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:32.327114 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.326844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:32.327114 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.326981 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:32.327114 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.327001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:32.327114 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.327034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:32.327114 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.327059 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:34.327038633 +0000 UTC m=+40.721678269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:32.327997 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.327965 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:38:32.328117 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.328039 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls podName:70d2e011-58cb-4646-adb9-04f8eacab25c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.328022541 +0000 UTC m=+46.722662186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls") pod "dns-default-xwtsr" (UID: "70d2e011-58cb-4646-adb9-04f8eacab25c") : secret "dns-default-metrics-tls" not found Oct 30 00:38:32.328117 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.328115 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:38:32.328233 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.328162 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert podName:abb2b77c-ea98-49f5-a77d-f531c74980c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:40.328150028 +0000 UTC m=+46.722789669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vsv6w" (UID: "abb2b77c-ea98-49f5-a77d-f531c74980c0") : secret "networking-console-plugin-cert" not found Oct 30 00:38:32.328292 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.328238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:32.328561 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.328430 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:32.328561 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.328496 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:34.328474958 +0000 UTC m=+40.723114607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:32.442718 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.442627 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk7xr" event={"ID":"0007937d-ef02-42e5-84de-6d9b8706537d","Type":"ContainerStarted","Data":"b060c76e367f9d2969f7a68d26bf2b849c9f5df70b5e9b95d5727fa4758c3af7"} Oct 30 00:38:32.443181 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.442774 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:32.444434 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.444399 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" event={"ID":"936fadee-4c5d-4ca7-b4b8-9d107c949645","Type":"ContainerStarted","Data":"4fd8c29adfd8e5e365a905941718837e5f1772d6ea1b87d02e740d9188328d95"} Oct 30 00:38:32.445797 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.445731 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" event={"ID":"d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5","Type":"ContainerStarted","Data":"fc89fc096922a4da0bcab717fa1fde6556060f2268d5d959ef7a8b26d5b88820"} Oct 30 00:38:32.448991 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.448953 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/1.log" Oct 30 00:38:32.449764 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.449747 2577 scope.go:122] "RemoveContainer" containerID="8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd" Oct 30 00:38:32.449930 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:32.449908 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lzlc8_openshift-console-operator(fdd0dfbe-2086-4370-8336-051395569bd8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" podUID="fdd0dfbe-2086-4370-8336-051395569bd8" Oct 30 00:38:32.558772 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.558707 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qk7xr" podStartSLOduration=34.017387147 podStartE2EDuration="37.558686693s" totalImagesPullingTime="3.541299546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:32.50169978 +0000 UTC m=+38.896339454" watchObservedRunningTime="2026-10-30 00:38:32.558686693 +0000 UTC m=+38.953326345" Oct 30 00:38:32.601656 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:32.601585 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" podStartSLOduration=315.977989364 podStartE2EDuration="5m22.601567343s" totalImagesPullingTime="6.623577979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:32.60071566 +0000 UTC m=+38.995355312" watchObservedRunningTime="2026-10-30 00:38:32.601567343 +0000 UTC m=+38.996206997" Oct 30 00:38:33.455103 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:33.455057 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" event={"ID":"d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5","Type":"ContainerStarted","Data":"2c814f0d70726f7edbf181b303c50a6bb8ff3b8231067cb19396b60baa5ee982"} Oct 30 00:38:33.511161 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:33.511089 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" podStartSLOduration=1.830166702 podStartE2EDuration="3.511072921s" totalImagesPullingTime="1.680906219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:33.510258369 +0000 UTC m=+39.904898021" watchObservedRunningTime="2026-10-30 00:38:33.511072921 +0000 UTC m=+39.905712574" Oct 30 00:38:34.348987 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:34.348948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:34.349149 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:34.349009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:34.349149 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:34.349086 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:34.349149 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:34.349102 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:34.349260 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:34.349161 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:38.349148511 +0000 UTC m=+44.743788158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:34.349260 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:34.349174 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:38.349168227 +0000 UTC m=+44.743807856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:34.899658 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:34.899616 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:34.900034 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:34.899683 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:34.900034 ip-10-0-130-171 kubenswrapper[2577]: I1030 00:38:34.900008 2577 scope.go:122] "RemoveContainer" containerID="8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd" Oct 30 00:38:34.900202 ip-10-0-130-171 kubenswrapper[2577]: E1030 00:38:34.900184 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lzlc8_openshift-console-operator(fdd0dfbe-2086-4370-8336-051395569bd8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" podUID="fdd0dfbe-2086-4370-8336-051395569bd8" Oct 30 00:38:34.993845 ip-10-0-130-171 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:38:35.002976 ip-10-0-130-171 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:38:35.003152 ip-10-0-130-171 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:38:35.003206 ip-10-0-130-171 systemd[1]: kubelet.service: Consumed 3.243s CPU time. Oct 30 00:38:35.016003 ip-10-0-130-171 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:38:35.102653 ip-10-0-130-171 kubenswrapper[4905]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:38:35.102653 ip-10-0-130-171 kubenswrapper[4905]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:38:35.102653 ip-10-0-130-171 kubenswrapper[4905]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:38:35.102653 ip-10-0-130-171 kubenswrapper[4905]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:38:35.104819 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104710 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.104819 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104819 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.104819 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104823 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104827 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104830 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104833 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104835 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104838 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104842 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104847 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104850 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104853 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104855 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104858 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104861 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104864 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104866 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104869 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104872 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104879 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104882 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.104918 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104885 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104887 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104890 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104893 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104895 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104898 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104901 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104904 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104907 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104909 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104912 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104915 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104917 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104920 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104923 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104925 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104928 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104930 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104934 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.105381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104936 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104939 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104941 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104944 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104946 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104949 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104951 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104954 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104956 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104958 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104961 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104963 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104966 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104969 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104971 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104974 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104976 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104979 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104982 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104985 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.105860 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104988 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104990 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104993 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104995 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.104997 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105000 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105002 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105006 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105010 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105013 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105017 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105019 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105023 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105025 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105028 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105031 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105034 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105036 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105039 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105041 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.106335 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105044 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105047 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105050 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105052 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105055 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105057 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105060 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105062 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105065 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105067 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105069 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105073 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105075 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105078 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105081 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105083 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105085 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105088 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105091 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.106817 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105093 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105096 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105099 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105102 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105104 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105107 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105110 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105912 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105920 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105923 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105925 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105928 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105930 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105933 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105935 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105938 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105941 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105943 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105946 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105948 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.107298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105951 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105953 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105956 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105958 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105961 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105963 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105967 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105970 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105972 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105975 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105977 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105980 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105982 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105987 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105991 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105994 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105997 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.105999 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106002 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.107840 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106010 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106013 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106016 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106018 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106020 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106023 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106025 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106028 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106030 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106032 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106035 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106038 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106040 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106042 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106045 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106047 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106051 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106055 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106057 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106060 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.108288 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106063 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106065 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106067 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106070 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106072 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106075 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106077 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106081 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106083 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106086 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106088 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106091 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106093 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106101 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106104 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106107 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106109 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106112 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106114 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106117 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.108794 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106119 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106121 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106124 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106126 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106129 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106131 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106133 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106136 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106139 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106142 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106145 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106147 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106150 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106152 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106154 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106157 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106159 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106162 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106164 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106166 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.109307 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106169 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106171 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106173 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106176 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106178 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106180 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106189 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106191 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106194 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106196 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106199 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106202 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106205 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106208 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106773 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106780 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106782 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106785 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106788 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106790 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.109814 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106793 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106799 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106801 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106804 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106807 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106809 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106812 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106814 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106817 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106819 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106822 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106825 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106828 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106830 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106833 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106835 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106838 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106840 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106843 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106846 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.110297 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106849 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106852 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106854 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106857 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106860 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106863 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106865 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106868 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106870 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106874 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106877 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106880 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106883 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106886 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106890 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106894 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106898 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106900 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.110854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106903 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106906 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106908 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106910 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106913 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106915 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106918 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106921 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106924 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106926 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106928 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106931 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106933 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106937 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106939 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106941 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106944 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106946 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106949 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.111285 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106951 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106953 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106956 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106958 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106960 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106963 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106965 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106968 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106971 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106973 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106975 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106978 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106980 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106983 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106985 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106988 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106990 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106992 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.106995 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107001 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.112021 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107004 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107006 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107009 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107011 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107013 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107016 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107018 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107021 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107023 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107026 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107028 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107031 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107033 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107036 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107038 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107041 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107044 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107046 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107048 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.112777 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107051 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107053 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107056 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107058 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107158 4905 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107172 4905 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107179 4905 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107183 4905 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107188 4905 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107191 4905 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107195 4905 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107200 4905 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107205 4905 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107208 4905 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107211 4905 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107215 4905 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107218 4905 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107221 4905 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107224 4905 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107227 4905 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107229 4905 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107233 4905 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107237 4905 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:38:35.113261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107240 4905 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107243 4905 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107246 4905 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107249 4905 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107253 4905 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107256 4905 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107259 4905 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107262 4905 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107265 4905 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107268 4905 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107271 4905 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107274 4905 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107277 4905 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107281 4905 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107284 4905 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107287 4905 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107290 4905 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107293 4905 flags.go:64] FLAG: --enable-server="true" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107295 4905 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107300 4905 flags.go:64] FLAG: --event-burst="100" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107304 4905 flags.go:64] FLAG: --event-qps="50" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107309 4905 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107312 4905 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107315 4905 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:38:35.113826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107319 4905 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107322 4905 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107325 4905 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107328 4905 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107331 4905 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107334 4905 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107338 4905 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107342 4905 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107345 4905 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107348 4905 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107350 4905 flags.go:64] FLAG: --feature-gates="" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107354 4905 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107357 4905 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107360 4905 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107363 4905 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107366 4905 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107369 4905 flags.go:64] FLAG: --help="false" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107372 4905 flags.go:64] FLAG: --hostname-override="ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107375 4905 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107378 4905 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107381 4905 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107384 4905 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107388 4905 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107390 4905 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:38:35.114470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107394 4905 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107396 4905 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107399 4905 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107402 4905 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107405 4905 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107409 4905 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107412 4905 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107415 4905 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107418 4905 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107421 4905 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107424 4905 flags.go:64] FLAG: --lock-file="" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107427 4905 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107430 4905 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107433 4905 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107440 4905 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107443 4905 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107446 4905 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107449 4905 flags.go:64] FLAG: --logging-format="text" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107452 4905 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107455 4905 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107458 4905 flags.go:64] FLAG: --manifest-url="" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107461 4905 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107465 4905 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107468 4905 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107472 4905 flags.go:64] FLAG: --max-pods="110" Oct 30 00:38:35.115045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107475 4905 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107478 4905 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107481 4905 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107484 4905 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107486 4905 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107489 4905 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107492 4905 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107500 4905 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107503 4905 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107506 4905 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107509 4905 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107512 4905 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107532 4905 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107536 4905 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107539 4905 flags.go:64] FLAG: --port="10250" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107542 4905 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107545 4905 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a50824c050e7716d" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107548 4905 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107551 4905 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107554 4905 flags.go:64] FLAG: --register-node="true" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107557 4905 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107562 4905 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107565 4905 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107568 4905 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107571 4905 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107575 4905 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:38:35.115698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107578 4905 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107581 4905 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107583 4905 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107586 4905 flags.go:64] FLAG: --runonce="false" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107589 4905 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107592 4905 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107595 4905 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107598 4905 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107600 4905 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107603 4905 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107607 4905 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107610 4905 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107613 4905 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107615 4905 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107618 4905 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107621 4905 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107624 4905 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107627 4905 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107631 4905 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107637 4905 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107640 4905 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107643 4905 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107647 4905 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107650 4905 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107653 4905 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:38:35.116294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107656 4905 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107659 4905 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107663 4905 flags.go:64] FLAG: --v="2" Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107667 4905 flags.go:64] FLAG: --version="false" Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107671 4905 flags.go:64] FLAG: --vmodule="" Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107676 4905 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.107680 4905 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107817 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107820 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107824 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107827 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107829 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107831 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107834 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107836 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107839 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107841 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107844 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107846 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107849 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107852 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.116912 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107855 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107857 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107860 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107862 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107865 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107868 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107870 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107872 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107875 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107877 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107879 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107882 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107884 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107888 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107890 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107893 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107895 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107897 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107901 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.117430 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107903 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107905 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107908 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107910 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107912 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107916 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107918 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107920 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107923 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107925 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107927 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107930 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107932 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107934 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107938 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107941 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107945 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107947 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.117906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107950 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107952 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107955 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107958 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107960 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107962 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107964 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107967 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107971 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107974 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107976 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107980 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107984 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107986 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107990 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107992 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107995 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107997 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.107999 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108002 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.118569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108004 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108006 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108008 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108011 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108013 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108016 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108018 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108021 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108024 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108026 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108029 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108031 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108033 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108036 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108038 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108040 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108043 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108045 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108047 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108050 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.119124 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108053 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108056 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108058 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108061 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108063 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108065 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108067 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108071 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108073 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108075 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108078 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108080 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108083 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108085 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108087 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.108111 4905 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:38:35.119830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108147 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108150 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108153 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108155 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108158 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108160 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108163 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108165 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108167 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108170 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108172 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108174 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108177 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108180 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108182 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108185 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108188 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108191 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108193 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.120298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108195 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108200 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108203 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108205 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108209 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108214 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108217 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108220 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108222 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108225 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108228 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108230 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108232 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108235 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108237 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108240 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108242 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108245 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108247 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.120775 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108251 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108254 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108256 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108259 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108261 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108264 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108266 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108268 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108271 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108273 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108277 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108280 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108282 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108285 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108290 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108293 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108296 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108300 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108304 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108307 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.121283 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108310 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108312 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108315 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108317 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108320 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108322 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108325 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108327 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108329 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108332 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108334 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108337 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108339 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108342 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108344 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108346 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108349 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108351 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108354 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108356 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.121909 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108358 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108361 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108364 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108367 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108369 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108371 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108375 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108378 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108381 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108383 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108385 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108388 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108391 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108393 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108397 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108399 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108402 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108404 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108406 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108409 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.122906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108411 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108414 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108416 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108418 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108421 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108423 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108425 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.108427 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.113892 4905 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.113911 4905 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.113936 4905 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.113941 4905 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114014 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114019 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114024 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114027 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114030 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114033 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114036 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.123382 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114039 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114042 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114045 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114047 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114050 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114053 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114057 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114061 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114064 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114067 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114070 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114072 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114075 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114077 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114080 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114082 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114085 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114088 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114090 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.123866 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114094 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114096 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114099 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114102 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114104 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114107 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114110 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114112 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114115 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114118 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114121 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114123 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114126 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114129 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114131 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114134 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114136 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114139 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114141 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.124305 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114144 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114147 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114150 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114153 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114155 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114159 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114162 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114165 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114168 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114170 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114173 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114175 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114178 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114180 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114183 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114185 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114188 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114190 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114193 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.124782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114195 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114198 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114200 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114203 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114205 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114208 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114210 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114213 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114216 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114218 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114221 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114223 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114226 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114228 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114230 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114233 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114236 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114238 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114241 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114243 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.125252 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114245 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114248 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114250 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114253 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114255 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114258 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114260 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114263 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114265 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114268 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114270 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114273 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114275 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114278 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114281 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114283 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114286 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114288 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114290 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114293 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.125929 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114295 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114306 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.114337 4905 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114492 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114499 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114501 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114504 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114507 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114511 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114531 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114536 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114539 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114542 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114544 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114547 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.126417 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114550 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114552 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114555 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114557 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114559 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114562 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114564 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114567 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114569 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114571 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114574 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114577 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114579 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114582 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114585 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114587 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114590 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114592 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114594 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.126790 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114598 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114601 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114604 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114606 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114609 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114612 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114615 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114618 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114620 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114623 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114626 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114628 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114630 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114633 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114635 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114637 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114640 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114643 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114645 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114647 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.127240 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114650 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114652 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114655 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114657 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114659 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114662 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114664 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114667 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114669 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114674 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114676 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114679 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114681 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114683 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114686 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114689 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114691 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114694 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114696 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.127773 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114699 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114705 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114707 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114710 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114712 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114715 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114719 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114722 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114725 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114727 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114730 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114732 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114735 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114737 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114739 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114742 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114744 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114747 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114749 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114751 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.128247 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114754 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114756 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114760 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114763 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114765 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114768 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114770 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114773 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114775 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114777 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114780 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114782 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114784 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114787 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114790 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114792 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.129084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.114817 4905 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114850 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114857 4905 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114861 4905 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114864 4905 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114866 4905 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114869 4905 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114871 4905 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114874 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114876 4905 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114879 4905 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114881 4905 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114883 4905 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114886 4905 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114888 4905 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114891 4905 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114893 4905 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114896 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114898 4905 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:38:35.129737 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114902 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114905 4905 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114907 4905 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114910 4905 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114913 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114915 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114918 4905 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114920 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114924 4905 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114928 4905 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114931 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114934 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114936 4905 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114939 4905 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114941 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114945 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114948 4905 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114950 4905 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114952 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:38:35.130286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114955 4905 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114957 4905 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114960 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114962 4905 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114964 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114967 4905 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114970 4905 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114972 4905 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114974 4905 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114977 4905 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114979 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114981 4905 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114984 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114987 4905 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114990 4905 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114992 4905 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114995 4905 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114997 4905 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.114999 4905 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115002 4905 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:38:35.130905 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115004 4905 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115007 4905 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115009 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115012 4905 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115014 4905 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115016 4905 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115019 4905 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115022 4905 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115026 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115028 4905 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115031 4905 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115033 4905 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115035 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115038 4905 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115040 4905 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115043 4905 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115046 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115049 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115051 4905 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115054 4905 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:38:35.131419 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115057 4905 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115059 4905 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115062 4905 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115065 4905 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115067 4905 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115071 4905 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115073 4905 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115076 4905 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115078 4905 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115081 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115083 4905 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115085 4905 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115088 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115090 4905 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115093 4905 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115095 4905 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115098 4905 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115100 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115103 4905 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115105 4905 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:38:35.131920 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115109 4905 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115112 4905 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115114 4905 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115117 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115119 4905 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115122 4905 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115124 4905 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115126 4905 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:35.115130 4905 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.115186 4905 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.116207 4905 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.116281 4905 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.116453 4905 server.go:1014] "Starting client certificate rotation" Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.116554 4905 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.116610 4905 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:32:54 +0000 UTC" deadline="2028-05-16 12:02:00.875496318 +0000 UTC" Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.116639 4905 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13547h23m25.758859482s" Oct 30 00:38:35.132381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.116907 4905 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:38:35.132809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.118028 4905 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:38:35.132809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.119579 4905 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:38:35.132809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.121428 4905 log.go:25] "Validated CRI v1 image API" Oct 30 00:38:35.132809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.122271 4905 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:38:35.132809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.124389 4905 fs.go:135] Filesystem UUIDs: map[514f3dcc-b0f2-4605-96b8-a95ec06dca35:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8523c6ee-9241-4a22-8213-9b41969544ed:/dev/nvme0n1p3] Oct 30 00:38:35.132946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.124409 4905 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3114d7b1f54d710ef44bf0c9fb469ca25f4d481e887be9f94d78e66b3d887044/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3114d7b1f54d710ef44bf0c9fb469ca25f4d481e887be9f94d78e66b3d887044/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51b4e1462bc7c4f63fa303067b49e27d6ff548dad1a1bfb58fe285b96823a98f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51b4e1462bc7c4f63fa303067b49e27d6ff548dad1a1bfb58fe285b96823a98f/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b6808f532f675366d24340b0359d1fb63229390502c572174b1801b42ee02d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b6808f532f675366d24340b0359d1fb63229390502c572174b1801b42ee02d3/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/720bf01b0fd486f50696f0f12652964caafc24584b8101a10485df447ffd8a01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/720bf01b0fd486f50696f0f12652964caafc24584b8101a10485df447ffd8a01/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76516f609addbf05d522d0cd242ca9d4e0a9bcaa17113e260f09ebc9ba6e115c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76516f609addbf05d522d0cd242ca9d4e0a9bcaa17113e260f09ebc9ba6e115c/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/807a8c559e2d5f2c65dc80104769fe6a67023faaed1326f157148c06c34d59ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/807a8c559e2d5f2c65dc80104769fe6a67023faaed1326f157148c06c34d59ea/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/818ed041536d1539c25505ddee02ccd10a8a9d134ccd71fbb5f4db088bb66a7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/818ed041536d1539c25505ddee02ccd10a8a9d134ccd71fbb5f4db088bb66a7f/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dd74a23ea8e383175471708ecbcd5b636e23fce31c8fa8c89fcac3e6976d91f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dd74a23ea8e383175471708ecbcd5b636e23fce31c8fa8c89fcac3e6976d91f/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ad6334276b714b82b4942ebbbf441cf4fcbe8a209faa05249d1ea0d06d24bd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ad6334276b714b82b4942ebbbf441cf4fcbe8a209faa05249d1ea0d06d24bd1/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a24e5e110cb085dcec6f33bb38ae7ced3d5e531b0660f605be2656810e0dffe4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a24e5e110cb085dcec6f33bb38ae7ced3d5e531b0660f605be2656810e0dffe4/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9079f58e0915c79c36d52a1162f76f7be696333eb3ae20fd0acff33ab10e218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9079f58e0915c79c36d52a1162f76f7be696333eb3ae20fd0acff33ab10e218/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab08e824cc7eeb6566146f9c4cc49fbe3bddde04c1419f27fc4def652e5cd551/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab08e824cc7eeb6566146f9c4cc49fbe3bddde04c1419f27fc4def652e5cd551/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6076c8eb82be2cf9bb3759a9239e64c5019cdd36d7379871e2d80838139f857/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6076c8eb82be2cf9bb3759a9239e64c5019cdd36d7379871e2d80838139f857/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c53f95c650962bcd2bdee87475747a1f40e183257be07217f16332b25201c23d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c53f95c650962bcd2bdee87475747a1f40e183257be07217f16332b25201c23d/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e08f16ff0bbc0ef7cdb346c41c236170072938f63ddf8419d9cce2339720c348/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e08f16ff0bbc0ef7cdb346c41c236170072938f63ddf8419d9cce2339720c348/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0d538f05b7b05060cac4bb736cc111723c8aea3f71b5b9940e5b650c556a6f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0d538f05b7b05060cac4bb736cc111723c8aea3f71b5b9940e5b650c556a6f5/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1a017f0554efd71a70812367c9a628df56e8782c334fbecb1c14f27d2820449/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1a017f0554efd71a70812367c9a628df56e8782c334fbecb1c14f27d2820449/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaea34fcbcaefa793c7b8bb4c29610f252afe5f0f7f8db55582825ba0fb2e97a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaea34fcbcaefa793c7b8bb4c29610f252afe5f0f7f8db55582825ba0fb2e97a/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f47a4c498bb558652049ef528b129b2e568f9e99518b2411763b2ca210685063/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f47a4c498bb558652049ef528b129b2e568f9e99518b2411763b2ca210685063/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f808938b92d09766bb6ef08c34f05523b6576dbd13a917a0b10ff06913c7ac1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f808938b92d09766bb6ef08c34f05523b6576dbd13a917a0b10ff06913c7ac1b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc89fc096922a4da0bcab717fa1fde6556060f2268d5d959ef7a8b26d5b88820/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc89fc096922a4da0bcab717fa1fde6556060f2268d5d959ef7a8b26d5b88820/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0007937d-ef02-42e5-84de-6d9b8706537d/volumes/kubernetes.io~projected/kube-api-access-gplz4:{mountpoint:/var/lib/kubelet/pods/0007937d-ef02-42e5-84de-6d9b8706537d/volumes/kubernetes.io~projected/kube-api-access-gplz4 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08aeeb89-9b10-41a0-a8bf-d07d7642ff66/volumes/kubernetes.io~projected/kube-api-access-ggc6f:{mountpoint:/var/lib/kubelet/pods/08aeeb89-9b10-41a0-a8bf-d07d7642ff66/volumes/kubernetes.io~projected/kube-api-access-ggc6f major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2968e47e-4347-42e8-8944-6415c8cef439/volumes/kubernetes.io~projected/kube-api-access-dbftv:{mountpoint:/var/lib/kubelet/pods/2968e47e-4347-42e8-8944-6415c8cef439/volumes/kubernetes.io~projected/kube-api-access-dbftv major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b542409-f192-467d-92a8-626445065e1b/volumes/kubernetes.io~projected/kube-api-access-xgjtf:{mountpoint:/var/lib/kubelet/pods/2b542409-f192-467d-92a8-626445065e1b/volumes/kubernetes.io~projected/kube-api-access-xgjtf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bc7448f-822d-4574-af12-0545a3513234/volumes/kubernetes.io~projected/kube-api-access-2jmcb:{mountpoint:/var/lib/kubelet/pods/3bc7448f-822d-4574-af12-0545a3513234/volumes/kubernetes.io~projected/kube-api-access-2jmcb major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c119ce3-3076-4976-8eba-9cc42b321dbc/volumes/kubernetes.io~projected/kube-api-access-clv6k:{mountpoint:/var/lib/kubelet/pods/3c119ce3-3076-4976-8eba-9cc42b321dbc/volumes/kubernetes.io~projected/kube-api-access-clv6k major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c119ce3-3076-4976-8eba-9cc42b321dbc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3c119ce3-3076-4976-8eba-9cc42b321dbc/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/447c7c04-4097-4065-9e32-ac6b6dc15706/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/447c7c04-4097-4065-9e32-ac6b6dc15706/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f621def-706c-43f8-afcc-a79f4441a7c0/volumes/kubernetes.io~projected/kube-api-access-9cths:{mountpoint:/var/lib/kubelet/pods/5f621def-706c-43f8-afcc-a79f4441a7c0/volumes/kubernetes.io~projected/kube-api-access-9cths major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6334ee19-9ae2-42e9-b704-39213b81e260/volumes/kubernetes.io~projected/kube-api-access-wsfn8:{mountpoint:/var/lib/kubelet/pods/6334ee19-9ae2-42e9-b704-39213b81e260/volumes/kubernetes.io~projected/kube-api-access-wsfn8 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70b55144-3ccd-406b-a9d2-6be38a04586e/volumes/kubernetes.io~projected/kube-api-access-xsg4l:{mountpoint:/var/lib/kubelet/pods/70b55144-3ccd-406b-a9d2-6be38a04586e/volumes/kubernetes.io~projected/kube-api-access-xsg4l major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d2e011-58cb-4646-adb9-04f8eacab25c/volumes/kubernetes.io~projected/kube-api-access-gnv77:{mountpoint:/var/lib/kubelet/pods/70d2e011-58cb-4646-adb9-04f8eacab25c/volumes/kubernetes.io~projected/kube-api-access-gnv77 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7849b028-2379-4a22-bee9-d1fa3544b3b4/volumes/kubernetes.io~projected/kube-api-access-fvtht:{mountpoint:/var/lib/kubelet/pods/7849b028-2379-4a22-bee9-d1fa3544b3b4/volumes/kubernetes.io~projected/kube-api-access-fvtht major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7997439e-9f1e-495d-bac8-6674d4f4a48a/volumes/kubernetes.io~projected/kube-api-access-lrsc4:{mountpoint:/var/lib/kubelet/pods/7997439e-9f1e-495d-bac8-6674d4f4a48a/volumes/kubernetes.io~projected/kube-api-access-lrsc4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b1ecc80-c5e0-40ac-b11c-957168289d3b/volumes/kubernetes.io~projected/kube-api-access-h48cx:{mountpoint:/var/lib/kubelet/pods/7b1ecc80-c5e0-40ac-b11c-957168289d3b/volumes/kubernetes.io~projected/kube-api-access-h48cx major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86bb8832-40cc-49f3-b7f2-247e47c55c18/volumes/kubernetes.io~projected/kube-api-access-qq2rp:{mountpoint:/var/lib/kubelet/pods/86bb8832-40cc-49f3-b7f2-247e47c55c18/volumes/kubernetes.io~projected/kube-api-access-qq2rp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86bb8832-40cc-49f3-b7f2-247e47c55c18/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/86bb8832-40cc-49f3-b7f2-247e47c55c18/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~projected/kube-api-access-bgzqh:{mountpoint:/var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~projected/kube-api-access-bgzqh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb/volumes/kubernetes.io~projected/kube-api-access-q8d8n:{mountpoint:/var/lib/kubelet/pods/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb/volumes/kubernetes.io~projected/kube-api-access-q8d8n major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/936fadee-4c5d-4ca7-b4b8-9d107c949645/volumes/kubernetes.io~projected/kube-api-access-86h2w:{mountpoint:/var/lib/kubelet/pods/936fadee-4c5d-4ca7-b4b8-9d107c949645/volumes/kubernetes.io~projected/kube-api-access-86h2w major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~projected/kube-api-access-4rp99:{mountpoint:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~projected/kube-api-access-4rp99 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaee817c-3a6d-46d6-8ae3-d0fd7417e978/volumes/kubernetes.io~projected/kube-api-access-6wjh5:{mountpoint:/var/lib/kubelet/pods/aaee817c-3a6d-46d6-8ae3-d0fd7417e978/volumes/kubernetes.io~projected/kube-api-access-6wjh5 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaee817c-3a6d-46d6-8ae3-d0fd7417e978/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/aaee817c-3a6d-46d6-8ae3-d0fd7417e978/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbfb585c-3327-4d80-b2c3-058e29b0c881/volumes/kubernetes.io~projected/kube-api-access-fjlr6:{mountpoint:/var/lib/kubelet/pods/cbfb585c-3327-4d80-b2c3-058e29b0c881/volumes/kubernetes.io~projected/kube-api-access-fjlr6 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbfb585c-3327-4d80-b2c3-058e29b0c881/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cbfb585c-3327-4d80-b2c3-058e29b0c881/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5/volumes/kubernetes.io~projected/kube-api-access-rj9fq:{mountpoint:/var/lib/kubelet/pods/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5/volumes/kubernetes.io~projected/kube-api-access-rj9fq major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~projected/kube-api-access-4z99q:{mountpoint:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~projected/kube-api-access-4z99q major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volumes/kubernetes.io~projected/kube-api-access-cbd7g:{mountpoint:/var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volumes/kubernetes.io~projected/kube-api-access-cbd7g major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd14a22e-0b4d-45eb-a25e-111817cfe34d/volumes/kubernetes.io~projected/kube-api-access-z6qbk:{mountpoint:/var/lib/kubelet/pods/dd14a22e-0b4d-45eb-a25e-111817cfe34d/volumes/kubernetes.io~projected/kube-api-access-z6qbk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e15e8be2-85a8-49a0-8e58-98d477c5ca4c/volumes/kubernetes.io~projected/kube-api-access-8j2vg:{mountpoint:/var/lib/kubelet/pods/e15e8be2-85a8-49a0-8e58-98d477c5ca4c/volumes/kubernetes.io~projected/kube-api-access-8j2vg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee3561da-40f3-40db-a14b-ad558c5236d7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ee3561da-40f3-40db-a14b-ad558c5236d7/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8053370-48fa-4a20-8865-96d61d33a0b4/volumes/kubernetes.io~projected/kube-api-access-ppw8v:{mountpoint:/var/lib/kubelet/pods/f8053370-48fa-4a20-8865-96d61d33a0b4/volumes/kubernetes.io~projected/kube-api-access-ppw8v major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~projected/kube-api-access-ndvfx:{mountpoint:/var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~projected/kube-api-access-ndvfx major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdd0dfbe-2086-4370-8336-051395569bd8/volumes/kubernetes.io~projected/kube-api-access-xktxl:{mountpoint:/var/lib/kubelet/pods/fdd0dfbe-2086-4370-8336-051395569bd8/volumes/kubernetes.io~projected/kube-api-access-xktxl major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdd0dfbe-2086-4370-8336-051395569bd8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fdd0dfbe-2086-4370-8336-051395569bd8/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/592872630c1d9b071f45efd9387d30b3e5355d8fb88959527c135deeea0b5b0e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/60d891b9820f494043c5cf38c8f9788f092424228cb7b4b10ac2e9902baf1f6f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3b843822d11f6857c04c3c64326a769883bb30694909a71c494ec4ea292fddbf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/dc7d3587e1a9c7230f5ac696881fd090515395570b543fdfd7955ea8351e5fc6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e958fe165ff8ad4b5ef0066c54f67a2131a07818a885d4d9c17e0f30443fbf6c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1f957ac863315b0462ca7a7d0ed7bd9d0c555eadc8adc161e172d6baef24922d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/74464ab0e0638e66a22b361b5defe1c280b147cdb2693b682dce771f88fe55b8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d73ff1710de3abb0aaff43215086e0de351f5bc789af8abca5c906bfe7683005/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/fc7c8c000c4aabb3db771f957f73da31f0b85b60355dc421bddfde54bc195249/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/d41161abf1c38e8239a97424129fdc8aaab85822d0a628a9fb08ddc977a64044/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/4eadfc300d4c63de0a63936178238ca184e3e348dfa1ce7c45b5ae4867ed84da/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/6bd0e982fab51192ca7e2b8e80e6aed8e87f37de727dd4d879e8cb805e97e487/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/51258b0d767a898098c0829fc3488c5c5b2c703fb83fd15878cd725b3ae0b18f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/e216d7287204db753cf3a7ce0fa649e77d34105262d9c8ebc784f1214c8689de/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/facb55ea3d4dae165f126b87b33f192a05f0f67dbb98749fa5941fd736448442/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3628a3f5d897dcb210b93eb75b6d16c7f1176778f09ecedb598337710bd5f4f6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7ea869efa4c88d5cf2dbfc6978afa0730ca834c39fc0130de0648ed2d2f002fb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e421cfd8b4bd0acb6b9063bd0e47dafac146a472a6a4e4928d18a3e18a7e4d6a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bba70780574eb6611be27f65f5b6e5d765e52f004292e37532c29ac800b402bc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/898e88327be8552f3dd303ca4efde091955538c980b846eebdaedafda65d8004/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1b2a13c2ddf1c6afa6e53cb6cb2029302c0817d5647540191741bd236ae9d8be/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/aab8857d78d730e2856af8cc96153eecc3e25b500d00f8edda504cee771a19bb/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/55b9d05efb6964723691dab5d164fb309799d5b26c9aa31694ec8fce1ad8a8eb/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/51a6341bef8df680cbdfbdd3301c6f4a7bb45d1116b0208c4469e4b01735caa9/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/c8171bfffc9528ab242c08bac64253d6fc7e90578e3b6acc6af999240bdb4762/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/49ce116edad6f35b2025b2a7a98843714a24ca7ef1ebfff8035500c8f8c95b6a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/fb5ba15943ed2dd133573ac1593e238aa1eebce962c203c7f714937c7811e532/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/336a26c5431b125433a51a3008124accb67f0a1dac867825a8a7a4d4f78c0c50/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e5a327946efe02c0592d666fa063404c363db05d0a1a26bd3208cd1bcb7bc9a9/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e52e666de51fd2bf04e0f403767aa3dce53c478022a926d74e4cfe1228b4a6c0/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/7d4d07e1f049e5abcecf5960d75bac7b0b9eaa8d09a2d8daadbcdbd8ebe79c65/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/415c295f7c151e04ce8dedd74d8954a60865d205914f8103aeeab78aef3d2785/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/666abe8ca54ed9bcd0938791896835b03d823a2246c9b94f427a69f85a585a15/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5cd11cbde2db05bd3e0368c989cf3f20f112de8f10cc346cdc63f17e7194cb94/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/ad64cceaed877a875c2af46e2b3fc6906f2683538aa0df92e121689be97b5146/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/85f4255ee251dfff0ba2118bbdc3ca7131159438d7a7c9b3ccd2dcff3005f9b7/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/754f595d5a207543e44a6e4263f9e5c49b506fb0ab3c4231f5e7e35783d0d045/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/fdb15a84c920bdfd00f5b1e6456fc7674d44ee6a5912ce1243987aae790d70c4/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d9951f3f1a50662bdd9dcae5d670e628aa7dfad30e313c5e6dc9e40409953f20/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/17259a9e9a932e083b50db149c872d71e3fb88b6f180f6d54942ecd4330c3617/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/eb4c3319315099cc4ea28af9ab5c1521c7a89fb3d2e9cb109f5d9e67013e6a99/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/a9f153b7f06813b39ea6085b427ef5607ab397c4269f4f01efea14bfa8da35ac/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/88d0e6a6e73221932ab8a80c8ffc6c448b20e5f9d122bfa026b5141ad9e1adcc/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/215ff8d2a4edf87177a294523101588de378a1b49708917314e91ca371d03743/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/b4383d19734909a5de10bb8ffa7d4c125cc1538b319a29137c60c064e65ee6f1/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/991c40462ee06c9aaf32146de25466e482f67144706530357556c0bb5d09e4c6/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/34a1ede95769a72b9746bbfff09b19af3837382e6eae340dd3c5fb4bcef43acf/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5c35de73e566ba32c693037705a6d75d9e9a6a0f1f6666665d7c37a6aaf7629c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/21e78599d460044898cabc919e28ce2110929916d53839294cf3d68a0707d43e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3efbfaa684c6ca9ed88e648be5bd74a6767c692f9d2af79064fc223e1f356392/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4204ac161cbafbf197c84bc603ea485d44862384f6de2ad201899fcba98682c0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/beca353cf17b548bab54e921883ddfaee07e0fea56a68ec469d956956cdb673e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/247014d87a413432f44a8d43b5a90736be07c5f70f3e49c5cde3d8055f73c4fa/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/26c315d62d83b6c6c5a1942150fd20136f635b7bf2c18ef446468dedd6200185/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1a8c48240364ca2cb2f04bcd2e8e47d1a8a54b2fed7c76a6e268e2c353248fcb/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1d846bc2fbb4db708fb912b79fece27f75e687b2609cdc15d1f8209b4fa2176f/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 00:38:35.133745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.133380 4905 manager.go:217] Machine: {Timestamp:2026-10-30 00:38:35.132912069 +0000 UTC m=+0.076085253 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199352 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec212968c3c1af74221e282fa2ae1902 SystemUUID:ec212968-c3c1-af74-221e-282fa2ae1902 BootID:fca3343b-3e94-4540-bff3-9f73b25346ef Filesystems:[{Device:/run/containers/storage/overlay-containers/3114d7b1f54d710ef44bf0c9fb469ca25f4d481e887be9f94d78e66b3d887044/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbfb585c-3327-4d80-b2c3-058e29b0c881/volumes/kubernetes.io~projected/kube-api-access-fjlr6 DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dd74a23ea8e383175471708ecbcd5b636e23fce31c8fa8c89fcac3e6976d91f/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c119ce3-3076-4976-8eba-9cc42b321dbc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70d2e011-58cb-4646-adb9-04f8eacab25c/volumes/kubernetes.io~projected/kube-api-access-gnv77 DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b6808f532f675366d24340b0359d1fb63229390502c572174b1801b42ee02d3/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee3561da-40f3-40db-a14b-ad558c5236d7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e15e8be2-85a8-49a0-8e58-98d477c5ca4c/volumes/kubernetes.io~projected/kube-api-access-8j2vg DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/807a8c559e2d5f2c65dc80104769fe6a67023faaed1326f157148c06c34d59ea/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bc7448f-822d-4574-af12-0545a3513234/volumes/kubernetes.io~projected/kube-api-access-2jmcb DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76516f609addbf05d522d0cd242ca9d4e0a9bcaa17113e260f09ebc9ba6e115c/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/818ed041536d1539c25505ddee02ccd10a8a9d134ccd71fbb5f4db088bb66a7f/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b1ecc80-c5e0-40ac-b11c-957168289d3b/volumes/kubernetes.io~projected/kube-api-access-h48cx DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8053370-48fa-4a20-8865-96d61d33a0b4/volumes/kubernetes.io~projected/kube-api-access-ppw8v DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volumes/kubernetes.io~projected/kube-api-access-cbd7g DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~projected/kube-api-access-ndvfx DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e08f16ff0bbc0ef7cdb346c41c236170072938f63ddf8419d9cce2339720c348/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7997439e-9f1e-495d-bac8-6674d4f4a48a/volumes/kubernetes.io~projected/kube-api-access-lrsc4 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~projected/kube-api-access-4rp99 DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c119ce3-3076-4976-8eba-9cc42b321dbc/volumes/kubernetes.io~projected/kube-api-access-clv6k DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/447c7c04-4097-4065-9e32-ac6b6dc15706/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc89fc096922a4da0bcab717fa1fde6556060f2268d5d959ef7a8b26d5b88820/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2b542409-f192-467d-92a8-626445065e1b/volumes/kubernetes.io~projected/kube-api-access-xgjtf DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aaee817c-3a6d-46d6-8ae3-d0fd7417e978/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0007937d-ef02-42e5-84de-6d9b8706537d/volumes/kubernetes.io~projected/kube-api-access-gplz4 DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb/volumes/kubernetes.io~projected/kube-api-access-q8d8n DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab08e824cc7eeb6566146f9c4cc49fbe3bddde04c1419f27fc4def652e5cd551/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70b55144-3ccd-406b-a9d2-6be38a04586e/volumes/kubernetes.io~projected/kube-api-access-xsg4l DeviceMajor:0 DeviceMinor:324 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aaee817c-3a6d-46d6-8ae3-d0fd7417e978/volumes/kubernetes.io~projected/kube-api-access-6wjh5 DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0d538f05b7b05060cac4bb736cc111723c8aea3f71b5b9940e5b650c556a6f5/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6076c8eb82be2cf9bb3759a9239e64c5019cdd36d7379871e2d80838139f857/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5/volumes/kubernetes.io~projected/kube-api-access-rj9fq DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~projected/kube-api-access-bgzqh DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaea34fcbcaefa793c7b8bb4c29610f252afe5f0f7f8db55582825ba0fb2e97a/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/86bb8832-40cc-49f3-b7f2-247e47c55c18/volumes/kubernetes.io~projected/kube-api-access-qq2rp DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a24e5e110cb085dcec6f33bb38ae7ced3d5e531b0660f605be2656810e0dffe4/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ad6334276b714b82b4942ebbbf441cf4fcbe8a209faa05249d1ea0d06d24bd1/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbfb585c-3327-4d80-b2c3-058e29b0c881/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f621def-706c-43f8-afcc-a79f4441a7c0/volumes/kubernetes.io~projected/kube-api-access-9cths DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/720bf01b0fd486f50696f0f12652964caafc24584b8101a10485df447ffd8a01/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c53f95c650962bcd2bdee87475747a1f40e183257be07217f16332b25201c23d/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdd0dfbe-2086-4370-8336-051395569bd8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd14a22e-0b4d-45eb-a25e-111817cfe34d/volumes/kubernetes.io~projected/kube-api-access-z6qbk DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f808938b92d09766bb6ef08c34f05523b6576dbd13a917a0b10ff06913c7ac1b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1a017f0554efd71a70812367c9a628df56e8782c334fbecb1c14f27d2820449/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2968e47e-4347-42e8-8944-6415c8cef439/volumes/kubernetes.io~projected/kube-api-access-dbftv DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9079f58e0915c79c36d52a1162f76f7be696333eb3ae20fd0acff33ab10e218/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86bb8832-40cc-49f3-b7f2-247e47c55c18/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6334ee19-9ae2-42e9-b704-39213b81e260/volumes/kubernetes.io~projected/kube-api-access-wsfn8 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08aeeb89-9b10-41a0-a8bf-d07d7642ff66/volumes/kubernetes.io~projected/kube-api-access-ggc6f DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fdd0dfbe-2086-4370-8336-051395569bd8/volumes/kubernetes.io~projected/kube-api-access-xktxl DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes/kubernetes.io~projected/kube-api-access-4z99q DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51b4e1462bc7c4f63fa303067b49e27d6ff548dad1a1bfb58fe285b96823a98f/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c300dc5-864d-4faf-ad58-807fb3813895/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3bbd76-9720-4475-8a2f-c8bee23c004b/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/936fadee-4c5d-4ca7-b4b8-9d107c949645/volumes/kubernetes.io~projected/kube-api-access-86h2w DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f47a4c498bb558652049ef528b129b2e568f9e99518b2411763b2ca210685063/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7849b028-2379-4a22-bee9-d1fa3544b3b4/volumes/kubernetes.io~projected/kube-api-access-fvtht DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:51b4e1462bc7c4f MacAddress:e6:f7:87:2e:45:88 Speed:10000 Mtu:8901} {Name:5b6808f532f6753 MacAddress:16:3c:2a:2a:09:53 Speed:10000 Mtu:8901} {Name:720bf01b0fd486f MacAddress:e2:88:1c:74:63:0c Speed:10000 Mtu:8901} {Name:76516f609addbf0 MacAddress:8a:ba:9a:0f:72:38 Speed:10000 Mtu:8901} {Name:a24e5e110cb085d MacAddress:7a:c2:d0:c6:8f:37 Speed:10000 Mtu:8901} {Name:a9079f58e0915c7 MacAddress:16:86:55:82:b6:47 Speed:10000 Mtu:8901} {Name:b6076c8eb82be2c MacAddress:8a:88:0c:85:e9:98 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fa:69:78:71 Speed:0 Mtu:9001} {Name:br-int MacAddress:32:4d:b6:57:19:65 Speed:0 Mtu:8901} {Name:c53f95c650962bc MacAddress:96:b5:bc:4a:ee:4e Speed:10000 Mtu:8901} {Name:e08f16ff0bbc0ef MacAddress:a2:e4:1c:9e:50:96 Speed:10000 Mtu:8901} {Name:e0d538f05b7b050 MacAddress:fa:cd:51:18:81:be Speed:10000 Mtu:8901} {Name:eaea34fcbcaefa7 MacAddress:5a:3d:77:ed:c1:b9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fa:69:78:71 Speed:0 Mtu:9001} {Name:f47a4c498bb5586 MacAddress:0e:f3:ab:e1:6b:5d Speed:10000 Mtu:8901} {Name:fc89fc096922a4d MacAddress:ba:62:29:f7:1c:e0 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:8a:a6:cb:25:0c:94 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:82:a3:31:b7:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:38:35.133922 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.133751 4905 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:38:35.133922 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.133802 4905 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:38:35.134153 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134118 4905 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:38:35.134294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134155 4905 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-171.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:38:35.134348 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134304 4905 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:38:35.134348 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134314 4905 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:38:35.134348 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134325 4905 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:38:35.134348 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134341 4905 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:38:35.134446 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134362 4905 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:38:35.134569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134559 4905 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:38:35.134688 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134680 4905 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:38:35.134725 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134716 4905 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:38:35.134753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134726 4905 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:38:35.134753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134736 4905 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:38:35.134753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134746 4905 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:38:35.134848 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.134754 4905 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:38:35.135464 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.135449 4905 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:38:35.135464 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.135461 4905 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:38:35.136139 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.136117 4905 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:38:35.138963 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.138946 4905 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:38:35.138963 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.138962 4905 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:38:35.139072 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139063 4905 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:38:35.139285 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139273 4905 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:38:35.139317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139303 4905 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:38:35.139431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139420 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:38:35.139465 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139441 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:38:35.139465 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139449 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:38:35.139465 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139458 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139466 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139475 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139481 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139487 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139493 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139499 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139507 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:38:35.139558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139540 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:38:35.139763 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139577 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:38:35.139763 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.139587 4905 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:38:35.143355 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.143340 4905 server.go:1263] "Started kubelet" Oct 30 00:38:35.143510 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.143438 4905 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:38:35.143576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.143561 4905 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:38:35.144086 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.143604 4905 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:38:35.144086 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.144002 4905 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:38:35.144110 ip-10-0-130-171 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:38:35.147843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.145614 4905 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:38:35.149968 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.149946 4905 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:38:35.150087 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.150033 4905 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:38:35.150161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.150107 4905 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:32:56 +0000 UTC" deadline="2028-04-17 17:58:35.675218971 +0000 UTC" Oct 30 00:38:35.151281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.150133 4905 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12857h20m0.525091634s" Oct 30 00:38:35.151373 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.150278 4905 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:38:35.151373 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.151310 4905 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:38:35.151470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.150297 4905 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:38:35.151470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.150993 4905 factory.go:55] Registering systemd factory Oct 30 00:38:35.151470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.151408 4905 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:38:35.152672 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.152654 4905 factory.go:153] Registering CRI-O factory Oct 30 00:38:35.152672 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.152673 4905 factory.go:223] Registration of the crio container factory successfully Oct 30 00:38:35.152825 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.152715 4905 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:38:35.152825 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.152732 4905 factory.go:103] Registering Raw factory Oct 30 00:38:35.152825 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.152742 4905 manager.go:1199] Started watching for new ooms in manager Oct 30 00:38:35.153112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.153100 4905 manager.go:319] Starting recovery of all containers Oct 30 00:38:35.154623 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154579 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f2b8428-edb2-419a-9766-c05a126a03ab" volumeName="kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token" seLinuxMountContext="" Oct 30 00:38:35.154623 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154622 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c" volumeName="kubernetes.io/projected/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-kube-api-access-cbd7g" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154633 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="447c7c04-4097-4065-9e32-ac6b6dc15706" volumeName="kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154641 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d2e011-58cb-4646-adb9-04f8eacab25c" volumeName="kubernetes.io/configmap/70d2e011-58cb-4646-adb9-04f8eacab25c-config-volume" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154649 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86bb8832-40cc-49f3-b7f2-247e47c55c18" volumeName="kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-service-ca-bundle" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154656 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2739e90-51e9-431f-911d-2b3bfa587d04" volumeName="kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154663 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3bbd76-9720-4475-8a2f-c8bee23c004b" volumeName="kubernetes.io/projected/fc3bbd76-9720-4475-8a2f-c8bee23c004b-kube-api-access-ndvfx" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154670 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b1ecc80-c5e0-40ac-b11c-957168289d3b" volumeName="kubernetes.io/projected/7b1ecc80-c5e0-40ac-b11c-957168289d3b-kube-api-access-h48cx" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154680 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c300dc5-864d-4faf-ad58-807fb3813895" volumeName="kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-tmp" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154686 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f2b8428-edb2-419a-9766-c05a126a03ab" volumeName="kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154694 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7849b028-2379-4a22-bee9-d1fa3544b3b4" volumeName="kubernetes.io/projected/7849b028-2379-4a22-bee9-d1fa3544b3b4-kube-api-access-fvtht" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154702 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86bb8832-40cc-49f3-b7f2-247e47c55c18" volumeName="kubernetes.io/projected/86bb8832-40cc-49f3-b7f2-247e47c55c18-kube-api-access-qq2rp" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154708 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2739e90-51e9-431f-911d-2b3bfa587d04" volumeName="kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154715 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e15e8be2-85a8-49a0-8e58-98d477c5ca4c" volumeName="kubernetes.io/projected/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-kube-api-access-8j2vg" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154733 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3bbd76-9720-4475-8a2f-c8bee23c004b" volumeName="kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-default-certificate" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154740 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c300dc5-864d-4faf-ad58-807fb3813895" volumeName="kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-etc-tuned" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154748 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c300dc5-864d-4faf-ad58-807fb3813895" volumeName="kubernetes.io/projected/8c300dc5-864d-4faf-ad58-807fb3813895-kube-api-access-bgzqh" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154756 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc7448f-822d-4574-af12-0545a3513234" volumeName="kubernetes.io/projected/3bc7448f-822d-4574-af12-0545a3513234-kube-api-access-2jmcb" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154763 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c119ce3-3076-4976-8eba-9cc42b321dbc" volumeName="kubernetes.io/projected/3c119ce3-3076-4976-8eba-9cc42b321dbc-kube-api-access-clv6k" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154770 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdd0dfbe-2086-4370-8336-051395569bd8" volumeName="kubernetes.io/projected/fdd0dfbe-2086-4370-8336-051395569bd8-kube-api-access-xktxl" seLinuxMountContext="" Oct 30 00:38:35.154773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154783 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbfb585c-3327-4d80-b2c3-058e29b0c881" volumeName="kubernetes.io/secret/cbfb585c-3327-4d80-b2c3-058e29b0c881-serving-cert" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154791 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b542409-f192-467d-92a8-626445065e1b" volumeName="kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154799 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c119ce3-3076-4976-8eba-9cc42b321dbc" volumeName="kubernetes.io/secret/3c119ce3-3076-4976-8eba-9cc42b321dbc-serving-cert" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154806 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d2e011-58cb-4646-adb9-04f8eacab25c" volumeName="kubernetes.io/empty-dir/70d2e011-58cb-4646-adb9-04f8eacab25c-tmp-dir" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154813 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb" volumeName="kubernetes.io/configmap/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-serviceca" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154821 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaee817c-3a6d-46d6-8ae3-d0fd7417e978" volumeName="kubernetes.io/secret/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-cert" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154828 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3561da-40f3-40db-a14b-ad558c5236d7" volumeName="kubernetes.io/secret/ee3561da-40f3-40db-a14b-ad558c5236d7-agent-certs" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154835 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3bbd76-9720-4475-8a2f-c8bee23c004b" volumeName="kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-stats-auth" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154846 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b542409-f192-467d-92a8-626445065e1b" volumeName="kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154854 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86bb8832-40cc-49f3-b7f2-247e47c55c18" volumeName="kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-tmp" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154861 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86bb8832-40cc-49f3-b7f2-247e47c55c18" volumeName="kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-snapshots" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154868 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f2b8428-edb2-419a-9766-c05a126a03ab" volumeName="kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154877 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5" volumeName="kubernetes.io/projected/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5-kube-api-access-rj9fq" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154883 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdd0dfbe-2086-4370-8336-051395569bd8" volumeName="kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-trusted-ca" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154908 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdd0dfbe-2086-4370-8336-051395569bd8" volumeName="kubernetes.io/secret/fdd0dfbe-2086-4370-8336-051395569bd8-serving-cert" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154916 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="936fadee-4c5d-4ca7-b4b8-9d107c949645" volumeName="kubernetes.io/projected/936fadee-4c5d-4ca7-b4b8-9d107c949645-kube-api-access-86h2w" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154923 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb2b77c-ea98-49f5-a77d-f531c74980c0" volumeName="kubernetes.io/configmap/abb2b77c-ea98-49f5-a77d-f531c74980c0-nginx-conf" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154930 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbfb585c-3327-4d80-b2c3-058e29b0c881" volumeName="kubernetes.io/projected/cbfb585c-3327-4d80-b2c3-058e29b0c881-kube-api-access-fjlr6" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154938 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c" volumeName="kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-config" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154946 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c" volumeName="kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154953 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3561da-40f3-40db-a14b-ad558c5236d7" volumeName="kubernetes.io/configmap/ee3561da-40f3-40db-a14b-ad558c5236d7-konnectivity-ca" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154960 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8053370-48fa-4a20-8865-96d61d33a0b4" volumeName="kubernetes.io/projected/f8053370-48fa-4a20-8865-96d61d33a0b4-kube-api-access-ppw8v" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154967 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d2e011-58cb-4646-adb9-04f8eacab25c" volumeName="kubernetes.io/projected/70d2e011-58cb-4646-adb9-04f8eacab25c-kube-api-access-gnv77" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154974 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7997439e-9f1e-495d-bac8-6674d4f4a48a" volumeName="kubernetes.io/projected/7997439e-9f1e-495d-bac8-6674d4f4a48a-kube-api-access-lrsc4" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154980 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2739e90-51e9-431f-911d-2b3bfa587d04" volumeName="kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154988 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdd0dfbe-2086-4370-8336-051395569bd8" volumeName="kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-config" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.154997 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7997439e-9f1e-495d-bac8-6674d4f4a48a" volumeName="kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-daemon-config" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155004 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaee817c-3a6d-46d6-8ae3-d0fd7417e978" volumeName="kubernetes.io/projected/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-kube-api-access-6wjh5" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155012 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd14a22e-0b4d-45eb-a25e-111817cfe34d" volumeName="kubernetes.io/projected/dd14a22e-0b4d-45eb-a25e-111817cfe34d-kube-api-access-z6qbk" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155019 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7849b028-2379-4a22-bee9-d1fa3544b3b4" volumeName="kubernetes.io/configmap/7849b028-2379-4a22-bee9-d1fa3544b3b4-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155026 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f2b8428-edb2-419a-9766-c05a126a03ab" volumeName="kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155033 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b542409-f192-467d-92a8-626445065e1b" volumeName="kubernetes.io/projected/2b542409-f192-467d-92a8-626445065e1b-kube-api-access-xgjtf" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155042 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6334ee19-9ae2-42e9-b704-39213b81e260" volumeName="kubernetes.io/projected/6334ee19-9ae2-42e9-b704-39213b81e260-kube-api-access-wsfn8" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155049 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2739e90-51e9-431f-911d-2b3bfa587d04" volumeName="kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155057 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c" volumeName="kubernetes.io/secret/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155065 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb" volumeName="kubernetes.io/projected/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-kube-api-access-q8d8n" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155078 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbfb585c-3327-4d80-b2c3-058e29b0c881" volumeName="kubernetes.io/configmap/cbfb585c-3327-4d80-b2c3-058e29b0c881-config" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155086 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2739e90-51e9-431f-911d-2b3bfa587d04" volumeName="kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155094 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd14a22e-0b4d-45eb-a25e-111817cfe34d" volumeName="kubernetes.io/empty-dir/dd14a22e-0b4d-45eb-a25e-111817cfe34d-tmp-dir" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155101 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2739e90-51e9-431f-911d-2b3bfa587d04" volumeName="kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155109 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0007937d-ef02-42e5-84de-6d9b8706537d" volumeName="kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155117 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2968e47e-4347-42e8-8944-6415c8cef439" volumeName="kubernetes.io/projected/2968e47e-4347-42e8-8944-6415c8cef439-kube-api-access-dbftv" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155125 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c119ce3-3076-4976-8eba-9cc42b321dbc" volumeName="kubernetes.io/configmap/3c119ce3-3076-4976-8eba-9cc42b321dbc-config" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155133 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f621def-706c-43f8-afcc-a79f4441a7c0" volumeName="kubernetes.io/configmap/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-config" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155141 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b55144-3ccd-406b-a9d2-6be38a04586e" volumeName="kubernetes.io/projected/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-api-access-xsg4l" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155161 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7997439e-9f1e-495d-bac8-6674d4f4a48a" volumeName="kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-cni-binary-copy" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155169 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c" volumeName="kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-env-overrides" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155176 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86bb8832-40cc-49f3-b7f2-247e47c55c18" volumeName="kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155184 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08aeeb89-9b10-41a0-a8bf-d07d7642ff66" volumeName="kubernetes.io/projected/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kube-api-access-ggc6f" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155190 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f621def-706c-43f8-afcc-a79f4441a7c0" volumeName="kubernetes.io/projected/5f621def-706c-43f8-afcc-a79f4441a7c0-kube-api-access-9cths" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155198 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f2b8428-edb2-419a-9766-c05a126a03ab" volumeName="kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155205 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f2b8428-edb2-419a-9766-c05a126a03ab" volumeName="kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155212 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f2b8428-edb2-419a-9766-c05a126a03ab" volumeName="kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155220 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2739e90-51e9-431f-911d-2b3bfa587d04" volumeName="kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155226 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8053370-48fa-4a20-8865-96d61d33a0b4" volumeName="kubernetes.io/configmap/f8053370-48fa-4a20-8865-96d61d33a0b4-telemetry-config" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155234 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b542409-f192-467d-92a8-626445065e1b" volumeName="kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-binary-copy" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155244 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b55144-3ccd-406b-a9d2-6be38a04586e" volumeName="kubernetes.io/empty-dir/70b55144-3ccd-406b-a9d2-6be38a04586e-data-volume" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155255 4905 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86bb8832-40cc-49f3-b7f2-247e47c55c18" volumeName="kubernetes.io/secret/86bb8832-40cc-49f3-b7f2-247e47c55c18-serving-cert" seLinuxMountContext="" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155264 4905 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.155270 4905 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:38:35.155624 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:35.150327 4905 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-171.ec2.internal\" not found" Oct 30 00:38:35.169750 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.169711 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/1.log" Oct 30 00:38:35.169924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.169802 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/1.log" Oct 30 00:38:35.181384 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.180701 4905 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:38:35.182478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.182283 4905 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:38:35.182478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.182317 4905 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:38:35.182478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.182337 4905 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:38:35.182478 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:35.182376 4905 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:38:35.191073 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.191040 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/1.log" Oct 30 00:38:35.191248 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.191081 4905 generic.go:356] "Generic (PLEG): container finished" podID="fdd0dfbe-2086-4370-8336-051395569bd8" containerID="8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd" exitCode=255 Oct 30 00:38:35.201885 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.201851 4905 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:38:35.202368 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.201929 4905 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:38:35.208680 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.208649 4905 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:38:35.222785 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.222740 4905 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:38:35.222939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.222801 4905 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:38:35.234643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.234607 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="b0637da489d4f3589f2351d190d68bf0795aa769f67c57ba2f8a733414b3a408" exitCode=0 Oct 30 00:38:35.234643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.234638 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="dba62d15d8bf638af0742def77ade22310f38b322ee351dbc998aef456afd2b4" exitCode=0 Oct 30 00:38:35.234643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.234647 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="605b38d68687b51fd1e7903572d1f5fa6e0585a27f3d95ed3661d0ce7be08c4d" exitCode=0 Oct 30 00:38:35.234913 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.234655 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="cb9cfa7cfa949aa5bcdb2299cd1d8c1cb7d268f482d3f29dee9f542c0cef4fd0" exitCode=0 Oct 30 00:38:35.234913 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.234665 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="f25c62338b9f6e0ceba70a51df57a096bd812cd032d9b49183ff815304b031c4" exitCode=0 Oct 30 00:38:35.234913 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.234672 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b542409-f192-467d-92a8-626445065e1b" containerID="f5f85dcf83718dd6254bf0e659578ad54b5dc3e1e9168c78cf87d8b9a247d628" exitCode=0 Oct 30 00:38:35.245659 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.245569 4905 generic.go:356] "Generic (PLEG): container finished" podID="e11e125d29b156d087bacedc1ce8a773" containerID="a158535e6e4f5b602cf7a0648e547c542e83fc3767a3497ed2c879b299ddc1f9" exitCode=0 Oct 30 00:38:35.254011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.253986 4905 manager.go:324] Recovery completed Oct 30 00:38:35.273959 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.273932 4905 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:38:35.273959 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.273953 4905 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:38:35.274156 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.273980 4905 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:38:35.274156 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274149 4905 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:38:35.274213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274164 4905 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:38:35.274213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274189 4905 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:38:35.274213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274196 4905 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:38:35.274213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274203 4905 policy_none.go:50] "Start" Oct 30 00:38:35.274213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274214 4905 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:38:35.274351 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274227 4905 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:38:35.274396 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274384 4905 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:38:35.274428 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274396 4905 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:38:35.274428 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.274403 4905 policy_none.go:44] "Start" Oct 30 00:38:35.279973 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.279935 4905 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:38:35.283076 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:35.283052 4905 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 00:38:35.283714 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.283702 4905 manager.go:341] "Starting Device Plugin manager" Oct 30 00:38:35.284338 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:35.283734 4905 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:38:35.284376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284344 4905 server.go:87] "Starting device plugin registration server" Oct 30 00:38:35.284642 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284629 4905 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:38:35.284681 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284645 4905 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:38:35.284748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284733 4905 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:38:35.284870 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284835 4905 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:38:35.284986 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284875 4905 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:38:35.284986 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284881 4905 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:38:35.285094 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.284934 4905 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:38:35.284865065Z","UUID":"1079a984-c64b-4d29-bb69-aea1eff00cda","Handler":null,"Name":"","Endpoint":""} Oct 30 00:38:35.286372 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.286350 4905 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:38:35.286467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.286380 4905 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:38:35.286467 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:35.286429 4905 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:38:35.385196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.385162 4905 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:38:35.386581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.386562 4905 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:38:35.386716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.386594 4905 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:38:35.386716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.386604 4905 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-171.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:38:35.386716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.386651 4905 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.424341 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.424304 4905 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.424555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.424405 4905 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.484066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.483974 4905 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal"] Oct 30 00:38:35.484723 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.484692 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" event={"ID":"92d13e825dfc8d10deb63823b2be0da4","Type":"ContainerStarted","Data":"576dfca443b02c7019b1553b2560df41925686d3e5a644bad32038461de7896c"} Oct 30 00:38:35.484806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.484729 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" event={"ID":"92d13e825dfc8d10deb63823b2be0da4","Type":"ContainerStarted","Data":"30e87824e2ef7dd0b3fa082a4183bf9d15f092da9cdd7cbbcf9560737fea5488"} Oct 30 00:38:35.484806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.484743 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" event={"ID":"e11e125d29b156d087bacedc1ce8a773","Type":"ContainerStarted","Data":"090a7cd61050c136a0e914777c01afb91a2c4f690f058a499ac1159b3b8f14fd"} Oct 30 00:38:35.484806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.484754 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" event={"ID":"e11e125d29b156d087bacedc1ce8a773","Type":"ContainerDied","Data":"a158535e6e4f5b602cf7a0648e547c542e83fc3767a3497ed2c879b299ddc1f9"} Oct 30 00:38:35.484806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.484771 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" event={"ID":"e11e125d29b156d087bacedc1ce8a773","Type":"ContainerStarted","Data":"49f18aeca99400405a1a906296394ae001299960cad87af4632eb10e0fcf3a1a"} Oct 30 00:38:35.484928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.484819 4905 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.484928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.484819 4905 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.513026 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.512963 4905 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:38:35.513026 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:35.513019 4905 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.513331 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.513315 4905 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:38:35.513362 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:35.513356 4905 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.603709 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.603676 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.603709 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.603710 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.603900 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.603730 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92d13e825dfc8d10deb63823b2be0da4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" (UID: \"92d13e825dfc8d10deb63823b2be0da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.704352 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.704313 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.704352 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.704349 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.704556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.704365 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92d13e825dfc8d10deb63823b2be0da4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" (UID: \"92d13e825dfc8d10deb63823b2be0da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.704556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.704413 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.704556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.704422 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11e125d29b156d087bacedc1ce8a773-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" (UID: \"e11e125d29b156d087bacedc1ce8a773\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:35.704556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:35.704473 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92d13e825dfc8d10deb63823b2be0da4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" (UID: \"92d13e825dfc8d10deb63823b2be0da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:38:36.135192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.135154 4905 apiserver.go:52] "Watching apiserver" Oct 30 00:38:36.194420 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.194389 4905 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:38:36.195157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.195124 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-bzs4r","kube-system/kas-connection-checker-685895c7c9-wl87c","kube-system/konnectivity-agent-8f8v5","kube-system/kas-connection-checker-685895c7c9-zbr9f","openshift-ingress-canary/ingress-canary-hwvx2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5","openshift-ingress/router-default-7844cf9dc7-sv5gn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal","openshift-cluster-node-tuning-operator/tuned-k7g8f","openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx","openshift-dns/dns-default-xwtsr","openshift-network-diagnostics/network-check-target-qk7xr","openshift-image-registry/image-registry-85b6df665-xt6bh","kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb","kube-system/global-pull-secret-syncer-vctgv","openshift-multus/multus-2lt89","kube-system/kas-connection-checker-685895c7c9-nzzvk","openshift-dns/node-resolver-qnlzg","openshift-multus/network-metrics-daemon-xwnkz","openshift-image-registry/node-ca-2j4gq","openshift-network-console/networking-console-plugin-86c478bb88-vsv6w","openshift-network-operator/iptables-alerter-4lhrd","openshift-image-registry/image-registry-bdfc8c748-74smf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv","openshift-insights/insights-operator-7c4986d449-zws9n","openshift-insights/insights-runtime-extractor-wkhms","openshift-ovn-kubernetes/ovnkube-node-wmvs5","openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2","openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp","openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz","openshift-console-operator/console-operator-7cd56b8dbc-lzlc8"] Oct 30 00:38:36.196391 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.196369 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.197990 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.197969 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.198278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.198257 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.199056 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.199032 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.199759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.199741 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:36.205165 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.205135 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:36.208384 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.208357 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.209350 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.209332 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.209915 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.209897 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.214424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.214404 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.214424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.214407 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.214964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.214951 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.214996 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.214982 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.215024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.214996 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.215024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.215010 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 00:38:36.215081 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.215060 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.215535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.215509 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 00:38:36.215659 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.215648 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 00:38:36.215810 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.215795 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 00:38:36.216067 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.216051 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4hm7n\"" Oct 30 00:38:36.216117 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.216106 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tswvk\"" Oct 30 00:38:36.249622 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.249593 4905 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:38:36.249759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.249683 4905 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:36.254220 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.254203 4905 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:38:36.280705 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.280625 4905 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:38:36.280705 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.280679 4905 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-171.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" Oct 30 00:38:36.284843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.284829 4905 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:38:36.284916 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.284871 4905 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-171.ec2.internal" Oct 30 00:38:36.307033 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307001 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-systemd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.307113 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307039 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbd7g\" (UniqueName: \"kubernetes.io/projected/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-kube-api-access-cbd7g\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.307113 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307064 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbftv\" (UniqueName: \"kubernetes.io/projected/2968e47e-4347-42e8-8944-6415c8cef439-kube-api-access-dbftv\") pod \"kas-connection-checker-685895c7c9-wl87c\" (UID: \"2968e47e-4347-42e8-8944-6415c8cef439\") " pod="kube-system/kas-connection-checker-685895c7c9-wl87c" Oct 30 00:38:36.307168 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307116 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovn-node-metrics-cert\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.307168 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307136 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-config\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.307236 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307172 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfn8\" (UniqueName: \"kubernetes.io/projected/6334ee19-9ae2-42e9-b704-39213b81e260-kube-api-access-wsfn8\") pod \"kas-connection-checker-685895c7c9-zbr9f\" (UID: \"6334ee19-9ae2-42e9-b704-39213b81e260\") " pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" Oct 30 00:38:36.307236 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307198 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-dbus\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.307236 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307222 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-kubelet\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.307369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307246 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsg4l\" (UniqueName: \"kubernetes.io/projected/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-api-access-xsg4l\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.307843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307790 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70d2e011-58cb-4646-adb9-04f8eacab25c-config-volume\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.307929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307868 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.307979 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307948 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-serviceca\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:38:36.308026 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.307998 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.308136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308051 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.308136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308073 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-host\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:38:36.308136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308095 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-systemd-units\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.308230 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308152 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:36.308230 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308193 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3561da-40f3-40db-a14b-ad558c5236d7-konnectivity-ca\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:36.308300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308235 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-var-lib-kubelet\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.308300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308260 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.308376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308297 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-default-certificate\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.308376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308349 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.308465 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308393 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-registration-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.308465 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308430 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggc6f\" (UniqueName: \"kubernetes.io/projected/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kube-api-access-ggc6f\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.308583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308459 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7849b028-2379-4a22-bee9-d1fa3544b3b4-iptables-alerter-script\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:38:36.308583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308544 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clv6k\" (UniqueName: \"kubernetes.io/projected/3c119ce3-3076-4976-8eba-9cc42b321dbc-kube-api-access-clv6k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:36.308670 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308586 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-system-cni-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.308714 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308684 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-bin\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.308768 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308723 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-modprobe-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.308814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308747 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-run\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.308814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308795 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-default-certificate\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.308897 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308817 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-tmp\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.308897 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308875 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-cnibin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.308964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308948 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.309007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308956 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-tmp\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.309007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.308995 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.309081 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309022 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-config\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.309081 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309060 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d2e011-58cb-4646-adb9-04f8eacab25c-tmp-dir\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.309173 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309133 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d2e011-58cb-4646-adb9-04f8eacab25c-tmp-dir\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.309173 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309128 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.309263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309175 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.309263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309192 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.309263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309211 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-tmp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.309263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309237 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-etc-kubernetes\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.309263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309256 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.309467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309261 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.309467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309298 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-tmp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.309467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309303 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd0dfbe-2086-4370-8336-051395569bd8-serving-cert\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.309467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309348 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-socket-dir-parent\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.309467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309453 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-netns\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.309708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309486 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmcb\" (UniqueName: \"kubernetes.io/projected/3bc7448f-822d-4574-af12-0545a3513234-kube-api-access-2jmcb\") pod \"volume-data-source-validator-58c9f5f46d-xjbn5\" (UID: \"3bc7448f-822d-4574-af12-0545a3513234\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" Oct 30 00:38:36.309708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309549 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.309708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309591 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.309708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309631 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-log-socket\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.309708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309667 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8053370-48fa-4a20-8865-96d61d33a0b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.309878 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309704 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-etc-selinux\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.309878 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309814 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.309878 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309846 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndvfx\" (UniqueName: \"kubernetes.io/projected/fc3bbd76-9720-4475-8a2f-c8bee23c004b-kube-api-access-ndvfx\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.309975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309888 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-netns\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.309975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.309932 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:36.310043 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310028 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-conf\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.310083 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310052 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-kubelet-config\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.310118 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310080 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b55144-3ccd-406b-a9d2-6be38a04586e-data-volume\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.310118 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310111 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8d8n\" (UniqueName: \"kubernetes.io/projected/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-kube-api-access-q8d8n\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:38:36.310175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310144 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qbk\" (UniqueName: \"kubernetes.io/projected/dd14a22e-0b4d-45eb-a25e-111817cfe34d-kube-api-access-z6qbk\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:38:36.310229 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310214 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbfb585c-3327-4d80-b2c3-058e29b0c881-config\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:36.310307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310280 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b55144-3ccd-406b-a9d2-6be38a04586e-data-volume\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.310364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310356 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-sys-fs\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.310410 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310376 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-multus\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.310410 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310404 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.310499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310423 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-service-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.310499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310441 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.310499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310456 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-systemd\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.310499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310487 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86h2w\" (UniqueName: \"kubernetes.io/projected/936fadee-4c5d-4ca7-b4b8-9d107c949645-kube-api-access-86h2w\") pod \"network-check-source-6f78d59f99-gr6hz\" (UID: \"936fadee-4c5d-4ca7-b4b8-9d107c949645\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" Oct 30 00:38:36.310706 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310539 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtht\" (UniqueName: \"kubernetes.io/projected/7849b028-2379-4a22-bee9-d1fa3544b3b4-kube-api-access-fvtht\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:38:36.310706 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310689 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-netd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.310799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310714 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbfb585c-3327-4d80-b2c3-058e29b0c881-serving-cert\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:36.310799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310737 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-sys\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.310799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310762 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48cx\" (UniqueName: \"kubernetes.io/projected/7b1ecc80-c5e0-40ac-b11c-957168289d3b-kube-api-access-h48cx\") pod \"kas-connection-checker-685895c7c9-nzzvk\" (UID: \"7b1ecc80-c5e0-40ac-b11c-957168289d3b\") " pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" Oct 30 00:38:36.310952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310889 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-slash\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.311010 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.310972 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.311059 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311023 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9fq\" (UniqueName: \"kubernetes.io/projected/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5-kube-api-access-rj9fq\") pod \"migrator-66df5d496b-skjm2\" (UID: \"d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" Oct 30 00:38:36.311116 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311066 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-etc-tuned\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.311116 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311093 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-conf-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.311204 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311160 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.311258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311202 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c300dc5-864d-4faf-ad58-807fb3813895-etc-tuned\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.311258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311213 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3561da-40f3-40db-a14b-ad558c5236d7-agent-certs\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:36.311338 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311258 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-multus-certs\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.311338 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311293 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-kubelet\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.311338 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311326 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-var-lib-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.311435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311369 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-ovn\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.311435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311408 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.311558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311436 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2rp\" (UniqueName: \"kubernetes.io/projected/86bb8832-40cc-49f3-b7f2-247e47c55c18-kube-api-access-qq2rp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.311558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311495 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-script-lib\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.311636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311575 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.311636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311601 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.311710 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311630 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-host\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.311710 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311695 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.311776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311734 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.311820 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311803 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.311850 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311840 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysconfig\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.311879 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311862 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrsc4\" (UniqueName: \"kubernetes.io/projected/7997439e-9f1e-495d-bac8-6674d4f4a48a-kube-api-access-lrsc4\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.311914 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311879 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.311949 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311908 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7849b028-2379-4a22-bee9-d1fa3544b3b4-host-slash\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:38:36.311990 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.311962 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c119ce3-3076-4976-8eba-9cc42b321dbc-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:36.312115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312101 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-cnibin\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.312149 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312124 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-daemon-config\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.312177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312154 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-snapshots\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.312217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312203 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rp99\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.312269 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312254 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-env-overrides\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.312300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312280 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-device-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.312342 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312303 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-trusted-ca\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.312342 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312280 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86bb8832-40cc-49f3-b7f2-247e47c55c18-snapshots\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.312407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312371 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktxl\" (UniqueName: \"kubernetes.io/projected/fdd0dfbe-2086-4370-8336-051395569bd8-kube-api-access-xktxl\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.312407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312390 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzqh\" (UniqueName: \"kubernetes.io/projected/8c300dc5-864d-4faf-ad58-807fb3813895-kube-api-access-bgzqh\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.312472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312408 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnv77\" (UniqueName: \"kubernetes.io/projected/70d2e011-58cb-4646-adb9-04f8eacab25c-kube-api-access-gnv77\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.312472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312430 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-stats-auth\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.312472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312452 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppw8v\" (UniqueName: \"kubernetes.io/projected/f8053370-48fa-4a20-8865-96d61d33a0b4-kube-api-access-ppw8v\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.312632 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312473 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-socket-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.312632 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312492 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cths\" (UniqueName: \"kubernetes.io/projected/5f621def-706c-43f8-afcc-a79f4441a7c0-kube-api-access-9cths\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.312632 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312547 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.312632 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312584 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.312806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312696 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjtf\" (UniqueName: \"kubernetes.io/projected/2b542409-f192-467d-92a8-626445065e1b-kube-api-access-xgjtf\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.312806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312740 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-stats-auth\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.312806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312786 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd14a22e-0b4d-45eb-a25e-111817cfe34d-tmp-dir\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:38:36.312917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312823 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-kubernetes\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.312917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312875 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd14a22e-0b4d-45eb-a25e-111817cfe34d-tmp-dir\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:38:36.313006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312942 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-system-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.313006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.312975 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-cni-binary-copy\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.313093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313011 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-etc-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.313093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313087 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd14a22e-0b4d-45eb-a25e-111817cfe34d-hosts-file\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:38:36.313157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313118 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c119ce3-3076-4976-8eba-9cc42b321dbc-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:36.313212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313192 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z99q\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.313300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313270 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.313414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313304 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-node-log\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.313414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313334 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-cert\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:36.313414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313391 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.313579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313414 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.313579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313471 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.313579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313498 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:36.313579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313545 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.313702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313593 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/abb2b77c-ea98-49f5-a77d-f531c74980c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:36.313702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313625 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.313702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313649 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-hostroot\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.313785 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313709 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.313785 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313754 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlr6\" (UniqueName: \"kubernetes.io/projected/cbfb585c-3327-4d80-b2c3-058e29b0c881-kube-api-access-fjlr6\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:36.313785 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313776 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-lib-modules\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.313868 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313809 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b55144-3ccd-406b-a9d2-6be38a04586e-crio-socket\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.313868 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313854 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86bb8832-40cc-49f3-b7f2-247e47c55c18-serving-cert\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.313920 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313871 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-k8s-cni-cncf-io\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.313948 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313920 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-bin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.313985 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313949 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.313985 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.313982 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2vg\" (UniqueName: \"kubernetes.io/projected/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-kube-api-access-8j2vg\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:36.314041 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314008 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjh5\" (UniqueName: \"kubernetes.io/projected/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-kube-api-access-6wjh5\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:36.314069 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314037 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.314279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314258 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.314614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314453 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:38:36.314614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314481 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.314614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314487 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.314614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314512 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.314614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314531 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.314614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314557 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.315376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314574 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.315452 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.315367 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:38:36.316194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.314577 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.316194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.315781 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 00:38:36.316194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.315904 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.316194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.315597 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:38:36.316194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.316101 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:38:36.316439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.316212 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.316439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.316410 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.315385 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.316710 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.316961 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.316967 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.316978 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317065 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317077 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.317130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317098 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.317529 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317229 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.317578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317507 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:38:36.317619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317604 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.318077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317715 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:38:36.318077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317752 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 00:38:36.318077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317771 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.318077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.317929 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.318311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318131 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:38:36.318311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318149 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.318413 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318354 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.318537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318499 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70d2e011-58cb-4646-adb9-04f8eacab25c-config-volume\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.318585 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318555 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.318677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318662 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.318751 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318734 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3561da-40f3-40db-a14b-ad558c5236d7-konnectivity-ca\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:36.318828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318807 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.318963 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.318949 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.319044 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319031 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.319217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319205 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 00:38:36.319264 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319256 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:38:36.319335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319324 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:38:36.319404 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319389 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.319457 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319430 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 00:38:36.319566 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319554 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:38:36.319628 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319390 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:36.319676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319628 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-serviceca\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:38:36.319722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319678 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-config\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.319833 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.319819 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.320218 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.320072 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.321095 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.321077 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-service-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.321167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.321144 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbfb585c-3327-4d80-b2c3-058e29b0c881-config\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:36.321246 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.321230 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.321973 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.321956 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.323000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.322982 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-env-overrides\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.323082 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.323029 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b542409-f192-467d-92a8-626445065e1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.323082 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.323062 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-daemon-config\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.323196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.323183 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7997439e-9f1e-495d-bac8-6674d4f4a48a-cni-binary-copy\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.324164 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.324148 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.324258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.324242 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c119ce3-3076-4976-8eba-9cc42b321dbc-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:36.324355 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.324331 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/abb2b77c-ea98-49f5-a77d-f531c74980c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:36.324392 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.324375 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.327504 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.327488 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-config\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.328424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.328405 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:38:36.328492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.328439 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:38:36.328492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.328464 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 00:38:36.328492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.328481 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:38:36.328807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.328791 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7849b028-2379-4a22-bee9-d1fa3544b3b4-iptables-alerter-script\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:38:36.328926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.328912 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 00:38:36.330085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.330069 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd0dfbe-2086-4370-8336-051395569bd8-serving-cert\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.331227 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.331213 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbfb585c-3327-4d80-b2c3-058e29b0c881-serving-cert\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:36.331320 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.331305 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9xms5\"" Oct 30 00:38:36.332442 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.332428 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovnkube-script-lib\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.333143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.333126 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tmln4\"" Oct 30 00:38:36.333215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.333162 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:38:36.333215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.333191 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:38:36.333274 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.333218 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:38:36.333307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.333297 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-drt9j\"" Oct 30 00:38:36.334665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.334651 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-cert\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:36.338620 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.338587 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 00:38:36.338690 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.338657 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:38:36.338779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.338764 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-ovn-node-metrics-cert\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.338904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.338888 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mxnln\"" Oct 30 00:38:36.338947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.338904 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 00:38:36.339018 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339007 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 00:38:36.339105 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339084 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 00:38:36.339325 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339272 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2jw8t\"" Oct 30 00:38:36.339325 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339292 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:38:36.339325 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339294 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-545pj\"" Oct 30 00:38:36.339514 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339327 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqzrd\"" Oct 30 00:38:36.339514 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339329 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 00:38:36.339514 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339405 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8h57b\"" Oct 30 00:38:36.339965 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339909 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 00:38:36.339965 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339934 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 00:38:36.339965 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339935 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 00:38:36.340136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339966 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h424n\"" Oct 30 00:38:36.340136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.339966 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tb2kx\"" Oct 30 00:38:36.340136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340069 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8053370-48fa-4a20-8865-96d61d33a0b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.340636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340614 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clv6k\" (UniqueName: \"kubernetes.io/projected/3c119ce3-3076-4976-8eba-9cc42b321dbc-kube-api-access-clv6k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:36.340708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340639 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbd7g\" (UniqueName: \"kubernetes.io/projected/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-kube-api-access-cbd7g\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.340708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340638 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsg4l\" (UniqueName: \"kubernetes.io/projected/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-api-access-xsg4l\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.340708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340639 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfn8\" (UniqueName: \"kubernetes.io/projected/6334ee19-9ae2-42e9-b704-39213b81e260-kube-api-access-wsfn8\") pod \"kas-connection-checker-685895c7c9-zbr9f\" (UID: \"6334ee19-9ae2-42e9-b704-39213b81e260\") " pod="kube-system/kas-connection-checker-685895c7c9-zbr9f" Oct 30 00:38:36.340824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340723 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggc6f\" (UniqueName: \"kubernetes.io/projected/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kube-api-access-ggc6f\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.340824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340743 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2rp\" (UniqueName: \"kubernetes.io/projected/86bb8832-40cc-49f3-b7f2-247e47c55c18-kube-api-access-qq2rp\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.340935 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340914 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmcb\" (UniqueName: \"kubernetes.io/projected/3bc7448f-822d-4574-af12-0545a3513234-kube-api-access-2jmcb\") pod \"volume-data-source-validator-58c9f5f46d-xjbn5\" (UID: \"3bc7448f-822d-4574-af12-0545a3513234\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xjbn5" Oct 30 00:38:36.340966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.340934 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.341089 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341071 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.341159 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341144 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-smcj2\"" Oct 30 00:38:36.341192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341182 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q2s6t\"" Oct 30 00:38:36.341240 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341223 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndvfx\" (UniqueName: \"kubernetes.io/projected/fc3bbd76-9720-4475-8a2f-c8bee23c004b-kube-api-access-ndvfx\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.341277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341231 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-44cn9\"" Oct 30 00:38:36.341307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341274 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7gr5j\"" Oct 30 00:38:36.341377 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341362 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-98rrr\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341594 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341610 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5s9q7\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341646 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341643 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341667 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jxbz5\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341673 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qbmx5\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341687 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-gsl54\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341669 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ckcc7\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341144 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bp5j\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341728 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t5wsq\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341732 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg5gj\"" Oct 30 00:38:36.341757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.341727 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k8zgn\"" Oct 30 00:38:36.342171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.342023 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9cwfv\"" Oct 30 00:38:36.342171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.342047 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 00:38:36.342171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.342055 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:38:36.342171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.342073 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 00:38:36.342171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.342161 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c119ce3-3076-4976-8eba-9cc42b321dbc-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-skktb\" (UID: \"3c119ce3-3076-4976-8eba-9cc42b321dbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" Oct 30 00:38:36.342437 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.342417 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbftv\" (UniqueName: \"kubernetes.io/projected/2968e47e-4347-42e8-8944-6415c8cef439-kube-api-access-dbftv\") pod \"kas-connection-checker-685895c7c9-wl87c\" (UID: \"2968e47e-4347-42e8-8944-6415c8cef439\") " pod="kube-system/kas-connection-checker-685895c7c9-wl87c" Oct 30 00:38:36.342673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.342659 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0dfbe-2086-4370-8336-051395569bd8-trusted-ca\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.344283 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.344267 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86bb8832-40cc-49f3-b7f2-247e47c55c18-serving-cert\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.344350 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.344292 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.344350 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.344326 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.344477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.344460 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86bb8832-40cc-49f3-b7f2-247e47c55c18-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zws9n\" (UID: \"86bb8832-40cc-49f3-b7f2-247e47c55c18\") " pod="openshift-insights/insights-operator-7c4986d449-zws9n" Oct 30 00:38:36.348573 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.348552 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.348813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.348794 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.349820 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.349802 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447c7c04-4097-4065-9e32-ac6b6dc15706-original-pull-secret\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.350185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.350168 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.350863 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.350848 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.352042 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.352028 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3561da-40f3-40db-a14b-ad558c5236d7-agent-certs\") pod \"konnectivity-agent-8f8v5\" (UID: \"ee3561da-40f3-40db-a14b-ad558c5236d7\") " pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:36.357048 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.357024 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qbk\" (UniqueName: \"kubernetes.io/projected/dd14a22e-0b4d-45eb-a25e-111817cfe34d-kube-api-access-z6qbk\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:38:36.357138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.357121 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppw8v\" (UniqueName: \"kubernetes.io/projected/f8053370-48fa-4a20-8865-96d61d33a0b4-kube-api-access-ppw8v\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.357321 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.357304 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnv77\" (UniqueName: \"kubernetes.io/projected/70d2e011-58cb-4646-adb9-04f8eacab25c-kube-api-access-gnv77\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.357380 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.357327 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9fq\" (UniqueName: \"kubernetes.io/projected/d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5-kube-api-access-rj9fq\") pod \"migrator-66df5d496b-skjm2\" (UID: \"d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skjm2" Oct 30 00:38:36.357445 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.357398 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rp99\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.357780 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.357764 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzqh\" (UniqueName: \"kubernetes.io/projected/8c300dc5-864d-4faf-ad58-807fb3813895-kube-api-access-bgzqh\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.359677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.359661 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrsc4\" (UniqueName: \"kubernetes.io/projected/7997439e-9f1e-495d-bac8-6674d4f4a48a-kube-api-access-lrsc4\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.359878 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.359863 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtht\" (UniqueName: \"kubernetes.io/projected/7849b028-2379-4a22-bee9-d1fa3544b3b4-kube-api-access-fvtht\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:38:36.360559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.360543 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48cx\" (UniqueName: \"kubernetes.io/projected/7b1ecc80-c5e0-40ac-b11c-957168289d3b-kube-api-access-h48cx\") pod \"kas-connection-checker-685895c7c9-nzzvk\" (UID: \"7b1ecc80-c5e0-40ac-b11c-957168289d3b\") " pod="kube-system/kas-connection-checker-685895c7c9-nzzvk" Oct 30 00:38:36.362939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.362914 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktxl\" (UniqueName: \"kubernetes.io/projected/fdd0dfbe-2086-4370-8336-051395569bd8-kube-api-access-xktxl\") pod \"console-operator-7cd56b8dbc-lzlc8\" (UID: \"fdd0dfbe-2086-4370-8336-051395569bd8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:36.362983 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.362970 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cths\" (UniqueName: \"kubernetes.io/projected/5f621def-706c-43f8-afcc-a79f4441a7c0-kube-api-access-9cths\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.363052 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.363031 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlr6\" (UniqueName: \"kubernetes.io/projected/cbfb585c-3327-4d80-b2c3-058e29b0c881-kube-api-access-fjlr6\") pod \"service-ca-operator-544d749c8c-bd6gp\" (UID: \"cbfb585c-3327-4d80-b2c3-058e29b0c881\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" Oct 30 00:38:36.363204 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.363189 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z99q\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.364193 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.364178 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2vg\" (UniqueName: \"kubernetes.io/projected/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-kube-api-access-8j2vg\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:36.364316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.364295 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjtf\" (UniqueName: \"kubernetes.io/projected/2b542409-f192-467d-92a8-626445065e1b-kube-api-access-xgjtf\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.364458 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.364438 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86h2w\" (UniqueName: \"kubernetes.io/projected/936fadee-4c5d-4ca7-b4b8-9d107c949645-kube-api-access-86h2w\") pod \"network-check-source-6f78d59f99-gr6hz\" (UID: \"936fadee-4c5d-4ca7-b4b8-9d107c949645\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gr6hz" Oct 30 00:38:36.364489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.364471 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjh5\" (UniqueName: \"kubernetes.io/projected/aaee817c-3a6d-46d6-8ae3-d0fd7417e978-kube-api-access-6wjh5\") pod \"ingress-canary-hwvx2\" (UID: \"aaee817c-3a6d-46d6-8ae3-d0fd7417e978\") " pod="openshift-ingress-canary/ingress-canary-hwvx2" Oct 30 00:38:36.364658 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.364645 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplz4\" (UniqueName: \"kubernetes.io/projected/0007937d-ef02-42e5-84de-6d9b8706537d-kube-api-access-gplz4\") pod \"network-check-target-qk7xr\" (UID: \"0007937d-ef02-42e5-84de-6d9b8706537d\") " pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:36.365273 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.365254 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8d8n\" (UniqueName: \"kubernetes.io/projected/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-kube-api-access-q8d8n\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:38:36.415056 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415019 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.415056 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415054 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415073 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415105 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-hostroot\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415122 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-lib-modules\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415137 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b55144-3ccd-406b-a9d2-6be38a04586e-crio-socket\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415156 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-k8s-cni-cncf-io\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415179 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-bin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415181 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-hostroot\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415178 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415222 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-bin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415228 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415251 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-systemd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415210 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-systemd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415228 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-k8s-cni-cncf-io\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415279 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-lib-modules\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415294 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b55144-3ccd-406b-a9d2-6be38a04586e-crio-socket\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415304 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-dbus\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415183 4905 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415319 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.915300024 +0000 UTC m=+1.858473215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415351 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-dbus\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415187 4905 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415394 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.915380504 +0000 UTC m=+1.858553675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : secret "metrics-daemon-secret" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415413 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-kubelet\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415446 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415455 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls podName:70d2e011-58cb-4646-adb9-04f8eacab25c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.915442087 +0000 UTC m=+1.858615258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls") pod "dns-default-xwtsr" (UID: "70d2e011-58cb-4646-adb9-04f8eacab25c") : secret "dns-default-metrics-tls" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415462 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-kubelet\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415480 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415499 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-host\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415510 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415540 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-systemd-units\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415564 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415566 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415580 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-var-lib-kubelet\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415582 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb-host\") pod \"node-ca-2j4gq\" (UID: \"90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb\") " pod="openshift-image-registry/node-ca-2j4gq" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415595 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415582 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-systemd-units\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415618 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-registration-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415630 4905 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415639 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-system-cni-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415665 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-registration-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415632 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-var-lib-kubelet\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415681 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-bin\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415688 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-system-cni-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415734 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-bin\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415678 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415761 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415692 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert podName:abb2b77c-ea98-49f5-a77d-f531c74980c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.915675441 +0000 UTC m=+1.858848628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vsv6w" (UID: "abb2b77c-ea98-49f5-a77d-f531c74980c0") : secret "networking-console-plugin-cert" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415807 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.915793126 +0000 UTC m=+1.858966302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:36.415813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415824 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-modprobe-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415847 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-run\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415885 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-cnibin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415908 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415921 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-run\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415932 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415950 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-cnibin\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415924 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-modprobe-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415964 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-etc-kubernetes\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.415979 4905 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415993 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415998 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-etc-kubernetes\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.415997 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416014 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.916003867 +0000 UTC m=+1.859177038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416029 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-socket-dir-parent\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416046 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-netns\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416059 4905 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416067 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-log-socket\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416078 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-netns\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416085 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-etc-selinux\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416103 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-log-socket\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416058 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-socket-dir-parent\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416101 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-netns\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416115 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.916103694 +0000 UTC m=+1.859276883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416150 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-run-netns\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416151 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-etc-selinux\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416156 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-conf\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416182 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-kubelet-config\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416201 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-sys-fs\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416234 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-sys-fs\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416247 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447c7c04-4097-4065-9e32-ac6b6dc15706-kubelet-config\") pod \"global-pull-secret-syncer-vctgv\" (UID: \"447c7c04-4097-4065-9e32-ac6b6dc15706\") " pod="kube-system/global-pull-secret-syncer-vctgv" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416270 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-multus\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416280 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-conf\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416288 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-systemd\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416311 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-netd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416321 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-var-lib-cni-multus\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416328 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-sys\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416343 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-slash\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416349 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-systemd\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416358 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416358 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-cni-netd\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416370 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-slash\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416387 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416392 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-sys\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416410 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-conf-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416433 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416453 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-multus-conf-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416464 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-multus-certs\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416471 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416490 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-kubelet\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416506 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-host-run-multus-certs\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416536 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-var-lib-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416539 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-host-kubelet\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416559 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-ovn\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416576 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416582 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-var-lib-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416594 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416606 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-ovn\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416617 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416639 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-host\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416657 4905 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416663 4905 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416686 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.916677828 +0000 UTC m=+1.859850998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416697 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysconfig\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416701 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.916692421 +0000 UTC m=+1.859865591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416662 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysconfig\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416719 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-host\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416742 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416752 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-sysctl-d\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416762 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7849b028-2379-4a22-bee9-d1fa3544b3b4-host-slash\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416787 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-cnibin\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416787 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7849b028-2379-4a22-bee9-d1fa3544b3b4-host-slash\") pod \"iptables-alerter-4lhrd\" (UID: \"7849b028-2379-4a22-bee9-d1fa3544b3b4\") " pod="openshift-network-operator/iptables-alerter-4lhrd" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416812 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-device-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416817 4905 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416822 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b542409-f192-467d-92a8-626445065e1b-cnibin\") pod \"multus-additional-cni-plugins-bzs4r\" (UID: \"2b542409-f192-467d-92a8-626445065e1b\") " pod="openshift-multus/multus-additional-cni-plugins-bzs4r" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416851 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.916844207 +0000 UTC m=+1.860017377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416857 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-device-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416873 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-socket-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416890 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416912 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-kubernetes\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416918 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08aeeb89-9b10-41a0-a8bf-d07d7642ff66-socket-dir\") pod \"aws-ebs-csi-driver-node-2ldlx\" (UID: \"08aeeb89-9b10-41a0-a8bf-d07d7642ff66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2ldlx" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416933 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-system-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416949 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-etc-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416967 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd14a22e-0b4d-45eb-a25e-111817cfe34d-hosts-file\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.416985 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:36.916976378 +0000 UTC m=+1.860149548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.416997 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.416966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.417006 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd14a22e-0b4d-45eb-a25e-111817cfe34d-hosts-file\") pod \"node-resolver-qnlzg\" (UID: \"dd14a22e-0b4d-45eb-a25e-111817cfe34d\") " pod="openshift-dns/node-resolver-qnlzg" Oct 30 00:38:36.419185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.417012 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-node-log\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.419185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.417019 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-etc-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.419185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.417024 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7997439e-9f1e-495d-bac8-6674d4f4a48a-system-cni-dir\") pod \"multus-2lt89\" (UID: \"7997439e-9f1e-495d-bac8-6674d4f4a48a\") " pod="openshift-multus/multus-2lt89" Oct 30 00:38:36.419185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.417038 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-node-log\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.419185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.417040 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c-run-openvswitch\") pod \"ovnkube-node-wmvs5\" (UID: \"d8e3d552-3b9e-4c1e-bc5c-b63204d1ce6c\") " pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:36.419185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.417062 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c300dc5-864d-4faf-ad58-807fb3813895-etc-kubernetes\") pod \"tuned-k7g8f\" (UID: \"8c300dc5-864d-4faf-ad58-807fb3813895\") " pod="openshift-cluster-node-tuning-operator/tuned-k7g8f" Oct 30 00:38:36.498944 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.498909 4905 scope.go:122] "RemoveContainer" containerID="8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd" Oct 30 00:38:36.921255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921159 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:36.921255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921215 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921264 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921290 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921321 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921335 4905 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921349 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921402 4905 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921408 4905 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921432 4905 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921442 4905 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921405 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921386768 +0000 UTC m=+2.864559954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : secret "metrics-daemon-secret" not found Oct 30 00:38:36.921489 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921407 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921506 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921563 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls podName:70d2e011-58cb-4646-adb9-04f8eacab25c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921536841 +0000 UTC m=+2.864710025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls") pod "dns-default-xwtsr" (UID: "70d2e011-58cb-4646-adb9-04f8eacab25c") : secret "dns-default-metrics-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921593 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921582062 +0000 UTC m=+2.864755232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921615 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert podName:abb2b77c-ea98-49f5-a77d-f531c74980c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921606101 +0000 UTC m=+2.864779271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vsv6w" (UID: "abb2b77c-ea98-49f5-a77d-f531c74980c0") : secret "networking-console-plugin-cert" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921635 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921627228 +0000 UTC m=+2.864800420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921656 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921648188 +0000 UTC m=+2.864821360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921701 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921725 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921748 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921790 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921817 4905 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921830 4905 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921833 4905 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921851 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921839372 +0000 UTC m=+2.865012545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:36.921834 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921871 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.92185762 +0000 UTC m=+2.865030807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921890 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:36.921889 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921898 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:36.922384 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921924 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921913183 +0000 UTC m=+2.865086353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:36.922384 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921937 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921931112 +0000 UTC m=+2.865104282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:36.922384 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:36.921947 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.921941562 +0000 UTC m=+2.865114732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:37.079195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.079165 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:37.147258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.147228 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnlzg_dd14a22e-0b4d-45eb-a25e-111817cfe34d/dns-node-resolver/0.log" Oct 30 00:38:37.258091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.258061 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:38:37.258537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.258506 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/1.log" Oct 30 00:38:37.258618 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.258559 4905 generic.go:356] "Generic (PLEG): container finished" podID="fdd0dfbe-2086-4370-8336-051395569bd8" containerID="a38db966257d74ca81e12bcc2427b71e678064823dc8c71dcc4d6f7201e442f7" exitCode=255 Oct 30 00:38:37.258684 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.258658 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" event={"ID":"fdd0dfbe-2086-4370-8336-051395569bd8","Type":"ContainerDied","Data":"a38db966257d74ca81e12bcc2427b71e678064823dc8c71dcc4d6f7201e442f7"} Oct 30 00:38:37.258782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.258752 4905 scope.go:122] "RemoveContainer" containerID="8bcc2ff0ae6fc14d882a7845b8f01ed5ba95bb520171dd5258f70dedd151dcfd" Oct 30 00:38:37.259448 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.259432 4905 scope.go:122] "RemoveContainer" containerID="a38db966257d74ca81e12bcc2427b71e678064823dc8c71dcc4d6f7201e442f7" Oct 30 00:38:37.259648 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.259629 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lzlc8_openshift-console-operator(fdd0dfbe-2086-4370-8336-051395569bd8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" podUID="fdd0dfbe-2086-4370-8336-051395569bd8" Oct 30 00:38:37.750273 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.750246 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2j4gq_90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb/node-ca/0.log" Oct 30 00:38:37.932896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.932857 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:37.932896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.932897 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.932926 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.932964 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933001 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933017 4905 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933024 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933087 4905 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933090 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933095 4905 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933107 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:37.933138 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933088 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933072854 +0000 UTC m=+4.876246040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933151 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933019 4905 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933193 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933218 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933201594 +0000 UTC m=+4.876374770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933241 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933234604 +0000 UTC m=+4.876407774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : secret "metrics-daemon-secret" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933254 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933247968 +0000 UTC m=+4.876421138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933280 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933274738 +0000 UTC m=+4.876447908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933291 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933285696 +0000 UTC m=+4.876458866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933304 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933341 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933376 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933423 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933438 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933453 4905 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933468 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933458264 +0000 UTC m=+4.876631463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933470 4905 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933487 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933476606 +0000 UTC m=+4.876649801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:37.933502 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:37.933509 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:39.933501161 +0000 UTC m=+4.876674331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:37.934120 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.933544 4905 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:38:37.935852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.935823 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70d2e011-58cb-4646-adb9-04f8eacab25c-metrics-tls\") pod \"dns-default-xwtsr\" (UID: \"70d2e011-58cb-4646-adb9-04f8eacab25c\") " pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:37.935963 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:37.935875 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abb2b77c-ea98-49f5-a77d-f531c74980c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vsv6w\" (UID: \"abb2b77c-ea98-49f5-a77d-f531c74980c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:38.000366 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.000286 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" Oct 30 00:38:38.010134 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.010109 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:38.100876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.100843 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:38:38.263214 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.263142 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:38:38.291845 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.291811 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vsv6w"] Oct 30 00:38:38.296269 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:38.296238 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb2b77c_ea98_49f5_a77d_f531c74980c0.slice/crio-297ef4730e767bbe9f4279d0804158d6edd46291bd7276c6c53c2eb0f5f6e0ef WatchSource:0}: Error finding container 297ef4730e767bbe9f4279d0804158d6edd46291bd7276c6c53c2eb0f5f6e0ef: Status 404 returned error can't find the container with id 297ef4730e767bbe9f4279d0804158d6edd46291bd7276c6c53c2eb0f5f6e0ef Oct 30 00:38:38.298209 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.298193 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:38:38.307353 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.307332 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xwtsr"] Oct 30 00:38:38.308478 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:38.308450 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d2e011_58cb_4646_adb9_04f8eacab25c.slice/crio-fef2e252fc293707d9ebb1c01fd9ac60e339ba5d6d978371e6bba859b2140208 WatchSource:0}: Error finding container fef2e252fc293707d9ebb1c01fd9ac60e339ba5d6d978371e6bba859b2140208: Status 404 returned error can't find the container with id fef2e252fc293707d9ebb1c01fd9ac60e339ba5d6d978371e6bba859b2140208 Oct 30 00:38:38.544131 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.544053 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwvx2_aaee817c-3a6d-46d6-8ae3-d0fd7417e978/serve-healthcheck-canary/0.log" Oct 30 00:38:38.559807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.559769 4905 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:38.560156 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.560142 4905 scope.go:122] "RemoveContainer" containerID="a38db966257d74ca81e12bcc2427b71e678064823dc8c71dcc4d6f7201e442f7" Oct 30 00:38:38.560345 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:38.560327 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lzlc8_openshift-console-operator(fdd0dfbe-2086-4370-8336-051395569bd8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" podUID="fdd0dfbe-2086-4370-8336-051395569bd8" Oct 30 00:38:38.952444 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:38.952365 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-skjm2_d03b11d9-a4a1-496a-b6ea-6bc8a11bbea5/migrator/0.log" Oct 30 00:38:39.268578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.268534 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" event={"ID":"abb2b77c-ea98-49f5-a77d-f531c74980c0","Type":"ContainerStarted","Data":"297ef4730e767bbe9f4279d0804158d6edd46291bd7276c6c53c2eb0f5f6e0ef"} Oct 30 00:38:39.270937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.270896 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xwtsr" event={"ID":"70d2e011-58cb-4646-adb9-04f8eacab25c","Type":"ContainerStarted","Data":"fef2e252fc293707d9ebb1c01fd9ac60e339ba5d6d978371e6bba859b2140208"} Oct 30 00:38:39.348301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.348272 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-skktb_3c119ce3-3076-4976-8eba-9cc42b321dbc/kube-storage-version-migrator-operator/0.log" Oct 30 00:38:39.551748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.551673 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:39.552154 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.552138 4905 scope.go:122] "RemoveContainer" containerID="a38db966257d74ca81e12bcc2427b71e678064823dc8c71dcc4d6f7201e442f7" Oct 30 00:38:39.552371 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.552348 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lzlc8_openshift-console-operator(fdd0dfbe-2086-4370-8336-051395569bd8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" podUID="fdd0dfbe-2086-4370-8336-051395569bd8" Oct 30 00:38:39.954634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954559 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:39.954634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954613 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:39.954817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954645 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:39.954817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954699 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:39.954817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954729 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:39.954817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954760 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:39.954817 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954731 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:39.954817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954797 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:39.954817 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954809 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b6df665-xt6bh: secret "image-registry-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954837 4905 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954869 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls podName:d2739e90-51e9-431f-911d-2b3bfa587d04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.954847852 +0000 UTC m=+8.898021035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls") pod "image-registry-85b6df665-xt6bh" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04") : secret "image-registry-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.954834 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954776 4905 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954879 4905 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954888 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls podName:5f621def-706c-43f8-afcc-a79f4441a7c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.954879216 +0000 UTC m=+8.898052387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ch2cv" (UID: "5f621def-706c-43f8-afcc-a79f4441a7c0") : secret "samples-operator-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954891 4905 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954909 4905 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdfc8c748-74smf: secret "image-registry-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954776 4905 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954787 4905 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954913 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.954905793 +0000 UTC m=+8.898078982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : secret "router-metrics-certs-default" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954985 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle podName:fc3bbd76-9720-4475-8a2f-c8bee23c004b nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.954976348 +0000 UTC m=+8.898149518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle") pod "router-default-7844cf9dc7-sv5gn" (UID: "fc3bbd76-9720-4475-8a2f-c8bee23c004b") : configmap references non-existent config key: service-ca.crt Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.954996 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.954990308 +0000 UTC m=+8.898163478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.955009 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.95500082 +0000 UTC m=+8.898173991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.955022 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls podName:9f2b8428-edb2-419a-9766-c05a126a03ab nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.955017234 +0000 UTC m=+8.898190405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls") pod "image-registry-bdfc8c748-74smf" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab") : secret "image-registry-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.955031 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.955026032 +0000 UTC m=+8.898199202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:39.955044 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:39.955111 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.955124 4905 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:38:39.955762 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:39.955148 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:43.955143255 +0000 UTC m=+8.898316425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : secret "metrics-daemon-secret" not found Oct 30 00:38:40.275581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:40.275547 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xwtsr" event={"ID":"70d2e011-58cb-4646-adb9-04f8eacab25c","Type":"ContainerStarted","Data":"8c1a5a33d16b3d8ba1245ac0fd9dedd541ec5debe04ef0d37967e1f0dd7b45b9"} Oct 30 00:38:40.276047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:40.275633 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xwtsr" event={"ID":"70d2e011-58cb-4646-adb9-04f8eacab25c","Type":"ContainerStarted","Data":"56583a9b9455ebe202681c9c2f29c0e6aae1a00e4d93b22aca51e9eab945f512"} Oct 30 00:38:40.276047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:40.275665 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:40.277094 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:40.277063 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-vsv6w" event={"ID":"abb2b77c-ea98-49f5-a77d-f531c74980c0","Type":"ContainerStarted","Data":"0424e9e66bebd7794a5214deecdf8976ee1f590188c627219b613f1dda4bcdea"} Oct 30 00:38:40.277361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:40.277339 4905 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f8v5_ee3561da-40f3-40db-a14b-ad558c5236d7/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:40.708577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:40.708495 4905 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f8v5_ee3561da-40f3-40db-a14b-ad558c5236d7/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:40.851982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:40.851952 4905 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f8v5_ee3561da-40f3-40db-a14b-ad558c5236d7/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:41.047217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:41.047187 4905 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f8v5_ee3561da-40f3-40db-a14b-ad558c5236d7/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:43.385279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.385250 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:43.385657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.385485 4905 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:38:43.531413 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.531382 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8f8v5" Oct 30 00:38:43.992745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.992718 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:43.992946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.992750 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:43.992946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.992770 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:43.992946 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.992830 4905 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:43.992946 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.992847 4905 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:38:43.992946 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.992868 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls podName:f8053370-48fa-4a20-8865-96d61d33a0b4 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:51.992856949 +0000 UTC m=+16.936030119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6xspg" (UID: "f8053370-48fa-4a20-8865-96d61d33a0b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:38:43.992946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.992880 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:43.992946 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.992902 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:51.992886323 +0000 UTC m=+16.936059529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : secret "insights-runtime-extractor-tls" not found Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.992950 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.992976 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.993023 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.993057 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.993086 4905 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.993127 4905 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.993088 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.993131 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs podName:e15e8be2-85a8-49a0-8e58-98d477c5ca4c nodeName:}" failed. No retries permitted until 2026-10-30 00:38:51.993118649 +0000 UTC m=+16.936291819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs") pod "network-metrics-daemon-xwnkz" (UID: "e15e8be2-85a8-49a0-8e58-98d477c5ca4c") : secret "metrics-daemon-secret" not found Oct 30 00:38:43.993295 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:38:43.993180 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm podName:70b55144-3ccd-406b-a9d2-6be38a04586e nodeName:}" failed. No retries permitted until 2026-10-30 00:38:51.993168858 +0000 UTC m=+16.936342034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkhms" (UID: "70b55144-3ccd-406b-a9d2-6be38a04586e") : configmap "kube-rbac-proxy" not found Oct 30 00:38:43.993901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.993684 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc3bbd76-9720-4475-8a2f-c8bee23c004b-service-ca-bundle\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:43.995345 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.995322 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f621def-706c-43f8-afcc-a79f4441a7c0-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ch2cv\" (UID: \"5f621def-706c-43f8-afcc-a79f4441a7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:43.995431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.995408 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"image-registry-bdfc8c748-74smf\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:43.995510 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.995488 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"image-registry-85b6df665-xt6bh\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:43.995564 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.995543 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bbd76-9720-4475-8a2f-c8bee23c004b-metrics-certs\") pod \"router-default-7844cf9dc7-sv5gn\" (UID: \"fc3bbd76-9720-4475-8a2f-c8bee23c004b\") " pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:43.997342 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.997323 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:43.998754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.998741 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" Oct 30 00:38:44.000003 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:43.999987 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:44.009009 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:44.008984 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:44.347999 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:44.347977 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7844cf9dc7-sv5gn"] Oct 30 00:38:44.349147 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:44.349125 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3bbd76_9720_4475_8a2f_c8bee23c004b.slice/crio-8807e88e2a53f466b9b81e17ac5121d73b85fb87de92e0dfe0f8bdc5958aebad WatchSource:0}: Error finding container 8807e88e2a53f466b9b81e17ac5121d73b85fb87de92e0dfe0f8bdc5958aebad: Status 404 returned error can't find the container with id 8807e88e2a53f466b9b81e17ac5121d73b85fb87de92e0dfe0f8bdc5958aebad Oct 30 00:38:44.369670 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:44.369647 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b6df665-xt6bh"] Oct 30 00:38:44.372543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:44.372510 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bdfc8c748-74smf"] Oct 30 00:38:44.377623 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:44.377595 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2739e90_51e9_431f_911d_2b3bfa587d04.slice/crio-20bd597c758726333a0e5598f6cce9302a307fc8bdced8084ffa371b59a60c2f WatchSource:0}: Error finding container 20bd597c758726333a0e5598f6cce9302a307fc8bdced8084ffa371b59a60c2f: Status 404 returned error can't find the container with id 20bd597c758726333a0e5598f6cce9302a307fc8bdced8084ffa371b59a60c2f Oct 30 00:38:44.377854 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:44.377828 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2b8428_edb2_419a_9766_c05a126a03ab.slice/crio-aede66273346fdd37b321b708e174d5bdaffa2634841b13c2a43bffba99793ee WatchSource:0}: Error finding container aede66273346fdd37b321b708e174d5bdaffa2634841b13c2a43bffba99793ee: Status 404 returned error can't find the container with id aede66273346fdd37b321b708e174d5bdaffa2634841b13c2a43bffba99793ee Oct 30 00:38:44.384335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:44.384316 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv"] Oct 30 00:38:45.293560 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.293502 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" event={"ID":"5f621def-706c-43f8-afcc-a79f4441a7c0","Type":"ContainerStarted","Data":"dac7756ec36ba7b259d0e7eff453f5bf9c1719fd91b31e0aabf536528aa378f0"} Oct 30 00:38:45.295276 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.295251 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" event={"ID":"fc3bbd76-9720-4475-8a2f-c8bee23c004b","Type":"ContainerStarted","Data":"65d8247837339ee402c5772fba3d26cf13199909ebf5d44df9a2b78c0d72dbc2"} Oct 30 00:38:45.295419 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.295293 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" event={"ID":"fc3bbd76-9720-4475-8a2f-c8bee23c004b","Type":"ContainerStarted","Data":"8807e88e2a53f466b9b81e17ac5121d73b85fb87de92e0dfe0f8bdc5958aebad"} Oct 30 00:38:45.296988 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.296959 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" event={"ID":"d2739e90-51e9-431f-911d-2b3bfa587d04","Type":"ContainerStarted","Data":"35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de"} Oct 30 00:38:45.297107 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.297009 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" event={"ID":"d2739e90-51e9-431f-911d-2b3bfa587d04","Type":"ContainerStarted","Data":"20bd597c758726333a0e5598f6cce9302a307fc8bdced8084ffa371b59a60c2f"} Oct 30 00:38:45.298801 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.298775 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" event={"ID":"9f2b8428-edb2-419a-9766-c05a126a03ab","Type":"ContainerStarted","Data":"32a87bf2f35b1e801cf20171ae957f1f45bf98bfa0761a036b8220cbbca6cb8a"} Oct 30 00:38:45.298928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.298817 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" event={"ID":"9f2b8428-edb2-419a-9766-c05a126a03ab","Type":"ContainerStarted","Data":"aede66273346fdd37b321b708e174d5bdaffa2634841b13c2a43bffba99793ee"} Oct 30 00:38:45.997907 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:45.997876 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:46.000326 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:46.000305 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:46.304854 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:46.304830 4905 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:38:46.305292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:46.304857 4905 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:38:46.305292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:46.304855 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" event={"ID":"5f621def-706c-43f8-afcc-a79f4441a7c0","Type":"ContainerStarted","Data":"9b1f2550e5bf348cc5c9713f9ef905cda81efff8f0f4055a65702a5596b5a72e"} Oct 30 00:38:46.305292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:46.304932 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ch2cv" event={"ID":"5f621def-706c-43f8-afcc-a79f4441a7c0","Type":"ContainerStarted","Data":"f7c439f653cc0794dc435b8014ade5ab7e459635fae114da6426706f60d5e4e8"} Oct 30 00:38:47.310306 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:47.310259 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:38:47.312209 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:47.312186 4905 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:38:49.183569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:49.183536 4905 scope.go:122] "RemoveContainer" containerID="a38db966257d74ca81e12bcc2427b71e678064823dc8c71dcc4d6f7201e442f7" Oct 30 00:38:49.202312 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:49.202279 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wmvs5" Oct 30 00:38:49.320884 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:49.320853 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:38:49.321047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:49.320928 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" event={"ID":"fdd0dfbe-2086-4370-8336-051395569bd8","Type":"ContainerStarted","Data":"d505f31755a347c263596645669344a364218bc94d3ae1c8eff40e792827fc9a"} Oct 30 00:38:49.321226 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:49.321203 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:49.322340 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:49.322307 4905 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-lzlc8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body= Oct 30 00:38:49.322464 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:49.322359 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" podUID="fdd0dfbe-2086-4370-8336-051395569bd8" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" Oct 30 00:38:50.283698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:50.283671 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xwtsr" Oct 30 00:38:50.329254 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:50.329229 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lzlc8" Oct 30 00:38:50.784868 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:50.784837 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b6df665-xt6bh"] Oct 30 00:38:51.857037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:51.856992 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:38:52.063188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.063153 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:52.063188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.063191 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:52.063426 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.063237 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:52.063426 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.063295 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:52.063918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.063894 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b55144-3ccd-406b-a9d2-6be38a04586e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:52.065824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.065802 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b55144-3ccd-406b-a9d2-6be38a04586e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhms\" (UID: \"70b55144-3ccd-406b-a9d2-6be38a04586e\") " pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:52.065890 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.065854 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8053370-48fa-4a20-8865-96d61d33a0b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6xspg\" (UID: \"f8053370-48fa-4a20-8865-96d61d33a0b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:52.065923 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.065888 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15e8be2-85a8-49a0-8e58-98d477c5ca4c-metrics-certs\") pod \"network-metrics-daemon-xwnkz\" (UID: \"e15e8be2-85a8-49a0-8e58-98d477c5ca4c\") " pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:52.098790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.098766 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" Oct 30 00:38:52.105488 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.105466 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnkz" Oct 30 00:38:52.110533 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.110498 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkhms" Oct 30 00:38:52.422307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.422273 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwnkz"] Oct 30 00:38:52.423325 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:52.423302 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15e8be2_85a8_49a0_8e58_98d477c5ca4c.slice/crio-003a73dbf2797606c98913a8d5b5cccb8fca6e1f966603dbde5d55ea4f7158dc WatchSource:0}: Error finding container 003a73dbf2797606c98913a8d5b5cccb8fca6e1f966603dbde5d55ea4f7158dc: Status 404 returned error can't find the container with id 003a73dbf2797606c98913a8d5b5cccb8fca6e1f966603dbde5d55ea4f7158dc Oct 30 00:38:52.428676 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:52.428646 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8053370_48fa_4a20_8865_96d61d33a0b4.slice/crio-f61880633d99429cba66b485e8325bb55505fdebab55620b52dc5fc6afc377e4 WatchSource:0}: Error finding container f61880633d99429cba66b485e8325bb55505fdebab55620b52dc5fc6afc377e4: Status 404 returned error can't find the container with id f61880633d99429cba66b485e8325bb55505fdebab55620b52dc5fc6afc377e4 Oct 30 00:38:52.436186 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.436163 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg"] Oct 30 00:38:52.437775 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:52.437759 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhms"] Oct 30 00:38:52.439162 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:38:52.439134 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b55144_3ccd_406b_a9d2_6be38a04586e.slice/crio-d64923fba60a4d001f66b6e2d3dda7e2d274f2a84b2431443d2dea824c0d24c8 WatchSource:0}: Error finding container d64923fba60a4d001f66b6e2d3dda7e2d274f2a84b2431443d2dea824c0d24c8: Status 404 returned error can't find the container with id d64923fba60a4d001f66b6e2d3dda7e2d274f2a84b2431443d2dea824c0d24c8 Oct 30 00:38:53.343123 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.343094 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhms" event={"ID":"70b55144-3ccd-406b-a9d2-6be38a04586e","Type":"ContainerStarted","Data":"3afb8291bbfd252e3780b00a3e2de1150bf4df6a938ba6676c34bd41f653797b"} Oct 30 00:38:53.343568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.343156 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhms" event={"ID":"70b55144-3ccd-406b-a9d2-6be38a04586e","Type":"ContainerStarted","Data":"df875173a0e795023358385739e3d40bdb72d45ff2f09f257b72ffa8fcecb16d"} Oct 30 00:38:53.343568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.343187 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhms" event={"ID":"70b55144-3ccd-406b-a9d2-6be38a04586e","Type":"ContainerStarted","Data":"d64923fba60a4d001f66b6e2d3dda7e2d274f2a84b2431443d2dea824c0d24c8"} Oct 30 00:38:53.344668 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.344644 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwnkz" event={"ID":"e15e8be2-85a8-49a0-8e58-98d477c5ca4c","Type":"ContainerStarted","Data":"003a73dbf2797606c98913a8d5b5cccb8fca6e1f966603dbde5d55ea4f7158dc"} Oct 30 00:38:53.345873 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.345826 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" event={"ID":"f8053370-48fa-4a20-8865-96d61d33a0b4","Type":"ContainerStarted","Data":"f61880633d99429cba66b485e8325bb55505fdebab55620b52dc5fc6afc377e4"} Oct 30 00:38:53.409344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.409315 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:53.409479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.409464 4905 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:38:53.410877 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:53.410855 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7844cf9dc7-sv5gn" Oct 30 00:38:54.352409 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:54.352369 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6xspg" event={"ID":"f8053370-48fa-4a20-8865-96d61d33a0b4","Type":"ContainerStarted","Data":"980b5a9127b6cce16eb5e59c5bc1a5c61c5beb89ef3dd8bfd100a4eead0af2bc"} Oct 30 00:38:54.354609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:54.354574 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwnkz" event={"ID":"e15e8be2-85a8-49a0-8e58-98d477c5ca4c","Type":"ContainerStarted","Data":"30485eae22833746867b36cab7554527a5334cdaedbd6b28fc9d5d75fa1b7d8a"} Oct 30 00:38:54.354732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:54.354632 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwnkz" event={"ID":"e15e8be2-85a8-49a0-8e58-98d477c5ca4c","Type":"ContainerStarted","Data":"764dbe5b1e2aa688993884f6eae29254af64bdbcbd10184962635b420a2d95ce"} Oct 30 00:38:55.360141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:38:55.360094 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhms" event={"ID":"70b55144-3ccd-406b-a9d2-6be38a04586e","Type":"ContainerStarted","Data":"df2bd12de06f9fb63a74206df45eb8770b34048654a14d05bb23bd7fd0c727d9"} Oct 30 00:39:00.790881 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:00.790851 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:39:01.862085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:01.862057 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:39:03.064238 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.064207 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-29t9j"] Oct 30 00:39:03.100922 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.100896 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.132554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.132506 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:39:03.132682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.132507 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:39:03.137145 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137118 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-root\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137149 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-metrics-client-ca\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137170 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-sys\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137188 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137248 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-textfile\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137379 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137366 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137410 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137392 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-wtmp\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137448 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137419 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9g75\" (UniqueName: \"kubernetes.io/projected/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-kube-api-access-d9g75\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.137448 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.137437 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-tls\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.208959 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.208923 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:39:03.209236 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.209218 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p7bx4\"" Oct 30 00:39:03.209354 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.209335 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:39:03.238347 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238317 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-textfile\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238506 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238354 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238506 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238379 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-wtmp\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238506 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238418 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9g75\" (UniqueName: \"kubernetes.io/projected/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-kube-api-access-d9g75\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238506 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238446 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-tls\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238506 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238482 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-root\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238552 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-metrics-client-ca\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238584 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-sys\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238599 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-root\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238613 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238639 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-sys\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238605 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-wtmp\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.238793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.238751 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-textfile\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.239133 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.239107 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.239252 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.239170 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-metrics-client-ca\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.241006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.240983 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.241093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.241037 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-node-exporter-tls\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.265861 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.265830 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9g75\" (UniqueName: \"kubernetes.io/projected/cf9434e3-f9aa-428a-83b2-3adb3518cf0b-kube-api-access-d9g75\") pod \"node-exporter-29t9j\" (UID: \"cf9434e3-f9aa-428a-83b2-3adb3518cf0b\") " pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.453261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:03.453184 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29t9j" Oct 30 00:39:03.493479 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:39:03.493446 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9434e3_f9aa_428a_83b2_3adb3518cf0b.slice/crio-93b9602590187891339a27f72a77c6f77471e89e2f00138fe4c68ad7e5c07b1b WatchSource:0}: Error finding container 93b9602590187891339a27f72a77c6f77471e89e2f00138fe4c68ad7e5c07b1b: Status 404 returned error can't find the container with id 93b9602590187891339a27f72a77c6f77471e89e2f00138fe4c68ad7e5c07b1b Oct 30 00:39:04.394978 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:04.394935 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29t9j" event={"ID":"cf9434e3-f9aa-428a-83b2-3adb3518cf0b","Type":"ContainerStarted","Data":"93b9602590187891339a27f72a77c6f77471e89e2f00138fe4c68ad7e5c07b1b"} Oct 30 00:39:05.401002 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:05.400954 4905 generic.go:356] "Generic (PLEG): container finished" podID="cf9434e3-f9aa-428a-83b2-3adb3518cf0b" containerID="d189be3ccabeaf5d5ca8f2a748d0d5b2717256f9582825142219e610d4f457d2" exitCode=0 Oct 30 00:39:05.401361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:05.401048 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29t9j" event={"ID":"cf9434e3-f9aa-428a-83b2-3adb3518cf0b","Type":"ContainerDied","Data":"d189be3ccabeaf5d5ca8f2a748d0d5b2717256f9582825142219e610d4f457d2"} Oct 30 00:39:06.406736 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:06.406700 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29t9j" event={"ID":"cf9434e3-f9aa-428a-83b2-3adb3518cf0b","Type":"ContainerStarted","Data":"a28ff00e55f254d561f3ceba38378e516009eec2dfe048220a5c24e575c17840"} Oct 30 00:39:06.407218 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:06.406790 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29t9j" event={"ID":"cf9434e3-f9aa-428a-83b2-3adb3518cf0b","Type":"ContainerStarted","Data":"2702fb103a71907c07f17fa74f6fb74716309d21210a1fa28ea9d11da4a7d14f"} Oct 30 00:39:06.466323 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:06.466273 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-29t9j" podStartSLOduration=2.111017258 podStartE2EDuration="3.466260681s" totalImagesPullingTime="1.272258337s" totalInitContainerRuntime="82.985086ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:06.464137474 +0000 UTC m=+31.407310675" watchObservedRunningTime="2026-10-30 00:39:06.466260681 +0000 UTC m=+31.409433871" Oct 30 00:39:08.103459 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:08.103426 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qk7xr" Oct 30 00:39:12.412976 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:39:12.412939 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:39:13.330108 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:13.330078 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bdfc8c748-74smf"] Oct 30 00:39:15.804814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:15.804731 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" podUID="d2739e90-51e9-431f-911d-2b3bfa587d04" containerName="registry" containerID="cri-o://35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de" gracePeriod=30 Oct 30 00:39:16.043985 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.043962 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:39:16.136120 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136050 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.136249 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136128 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.136249 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136165 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.136249 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136188 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.136415 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136373 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.136464 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136429 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.136511 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136463 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.136589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.136567 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\") pod \"d2739e90-51e9-431f-911d-2b3bfa587d04\" (UID: \"d2739e90-51e9-431f-911d-2b3bfa587d04\") " Oct 30 00:39:16.138892 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.138747 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:16.139238 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.139073 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:16.139883 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.139404 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:16.139883 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.139552 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "kube-api-access-4z99q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:16.140035 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.139978 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:16.140035 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.139991 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:16.145224 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.145201 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:16.145797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.145767 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted" pod "d2739e90-51e9-431f-911d-2b3bfa587d04" (UID: "d2739e90-51e9-431f-911d-2b3bfa587d04"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:16.238685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238661 4905 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-certificates\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.238685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238683 4905 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-image-registry-private-configuration\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.238811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238692 4905 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2739e90-51e9-431f-911d-2b3bfa587d04-ca-trust-extracted\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.238811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238701 4905 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2739e90-51e9-431f-911d-2b3bfa587d04-installation-pull-secrets\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.238811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238709 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z99q\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-kube-api-access-4z99q\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.238811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238735 4905 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-bound-sa-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.238811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238744 4905 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2739e90-51e9-431f-911d-2b3bfa587d04-registry-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.238811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.238751 4905 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2739e90-51e9-431f-911d-2b3bfa587d04-trusted-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:16.442025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.441971 4905 generic.go:356] "Generic (PLEG): container finished" podID="d2739e90-51e9-431f-911d-2b3bfa587d04" containerID="35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de" exitCode=0 Oct 30 00:39:16.442104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.442024 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" Oct 30 00:39:16.442104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.442048 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" event={"ID":"d2739e90-51e9-431f-911d-2b3bfa587d04","Type":"ContainerDied","Data":"35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de"} Oct 30 00:39:16.442104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.442086 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b6df665-xt6bh" event={"ID":"d2739e90-51e9-431f-911d-2b3bfa587d04","Type":"ContainerDied","Data":"20bd597c758726333a0e5598f6cce9302a307fc8bdced8084ffa371b59a60c2f"} Oct 30 00:39:16.442197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.442109 4905 scope.go:122] "RemoveContainer" containerID="35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de" Oct 30 00:39:16.450047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.450030 4905 scope.go:122] "RemoveContainer" containerID="35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de" Oct 30 00:39:16.450288 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:39:16.450269 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de\": container with ID starting with 35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de not found: ID does not exist" containerID="35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de" Oct 30 00:39:16.450353 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.450296 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de"} err="failed to get container status \"35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de\": rpc error: code = NotFound desc = could not find container \"35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de\": container with ID starting with 35f976debdc614dc0984ac6b812a61e36ffa1a378d22bd9c020d068c986633de not found: ID does not exist" Oct 30 00:39:16.495639 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.495615 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b6df665-xt6bh"] Oct 30 00:39:16.509243 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:16.509204 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85b6df665-xt6bh"] Oct 30 00:39:17.187820 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:17.187788 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2739e90-51e9-431f-911d-2b3bfa587d04" path="/var/lib/kubelet/pods/d2739e90-51e9-431f-911d-2b3bfa587d04/volumes" Oct 30 00:39:38.349656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.349619 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" podUID="9f2b8428-edb2-419a-9766-c05a126a03ab" containerName="registry" containerID="cri-o://32a87bf2f35b1e801cf20171ae957f1f45bf98bfa0761a036b8220cbbca6cb8a" gracePeriod=30 Oct 30 00:39:38.530238 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.530203 4905 generic.go:356] "Generic (PLEG): container finished" podID="9f2b8428-edb2-419a-9766-c05a126a03ab" containerID="32a87bf2f35b1e801cf20171ae957f1f45bf98bfa0761a036b8220cbbca6cb8a" exitCode=0 Oct 30 00:39:38.530379 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.530276 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" event={"ID":"9f2b8428-edb2-419a-9766-c05a126a03ab","Type":"ContainerDied","Data":"32a87bf2f35b1e801cf20171ae957f1f45bf98bfa0761a036b8220cbbca6cb8a"} Oct 30 00:39:38.575876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.575857 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:39:38.690949 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.690890 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.690949 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.690930 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.691118 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.690955 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.691118 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691094 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.691228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691157 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.691228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691192 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.691324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691250 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.691324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691263 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:38.691324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691289 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") pod \"9f2b8428-edb2-419a-9766-c05a126a03ab\" (UID: \"9f2b8428-edb2-419a-9766-c05a126a03ab\") " Oct 30 00:39:38.691534 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691491 4905 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-trusted-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.691831 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.691738 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:38.693422 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.693386 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:38.693556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.693474 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:38.693652 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.693631 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "kube-api-access-4rp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:38.694102 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.694071 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:38.694188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.694139 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:38.698044 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.698025 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted" pod "9f2b8428-edb2-419a-9766-c05a126a03ab" (UID: "9f2b8428-edb2-419a-9766-c05a126a03ab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:38.791992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.791962 4905 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-image-registry-private-configuration\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.791992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.791991 4905 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-bound-sa-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.792112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.792002 4905 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f2b8428-edb2-419a-9766-c05a126a03ab-ca-trust-extracted\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.792112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.792010 4905 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-certificates\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.792112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.792019 4905 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-registry-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.792112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.792027 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rp99\" (UniqueName: \"kubernetes.io/projected/9f2b8428-edb2-419a-9766-c05a126a03ab-kube-api-access-4rp99\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.792112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:38.792035 4905 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f2b8428-edb2-419a-9766-c05a126a03ab-installation-pull-secrets\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:39:39.534849 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:39.534822 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" Oct 30 00:39:39.535196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:39.534858 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdfc8c748-74smf" event={"ID":"9f2b8428-edb2-419a-9766-c05a126a03ab","Type":"ContainerDied","Data":"aede66273346fdd37b321b708e174d5bdaffa2634841b13c2a43bffba99793ee"} Oct 30 00:39:39.535196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:39.534914 4905 scope.go:122] "RemoveContainer" containerID="32a87bf2f35b1e801cf20171ae957f1f45bf98bfa0761a036b8220cbbca6cb8a" Oct 30 00:39:39.602411 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:39.602379 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bdfc8c748-74smf"] Oct 30 00:39:39.616700 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:39.616677 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bdfc8c748-74smf"] Oct 30 00:39:41.187163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:41.187132 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2b8428-edb2-419a-9766-c05a126a03ab" path="/var/lib/kubelet/pods/9f2b8428-edb2-419a-9766-c05a126a03ab/volumes" Oct 30 00:39:42.412254 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:39:42.412220 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:39:46.562406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:46.562370 4905 generic.go:356] "Generic (PLEG): container finished" podID="cbfb585c-3327-4d80-b2c3-058e29b0c881" containerID="b6741522c805278b29fd5009314cff26ebff1b6ee965d54a2d6ff50c418590ee" exitCode=0 Oct 30 00:39:46.562755 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:46.562422 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" event={"ID":"cbfb585c-3327-4d80-b2c3-058e29b0c881","Type":"ContainerDied","Data":"b6741522c805278b29fd5009314cff26ebff1b6ee965d54a2d6ff50c418590ee"} Oct 30 00:39:46.562755 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:46.562733 4905 scope.go:122] "RemoveContainer" containerID="b6741522c805278b29fd5009314cff26ebff1b6ee965d54a2d6ff50c418590ee" Oct 30 00:39:47.567715 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:47.567680 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bd6gp" event={"ID":"cbfb585c-3327-4d80-b2c3-058e29b0c881","Type":"ContainerStarted","Data":"dbd0946314dea7e4648fdd537e860c5fa7816f3e9dcfa1a5de5b9ba98b898737"} Oct 30 00:39:55.596380 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:55.596351 4905 generic.go:356] "Generic (PLEG): container finished" podID="3c119ce3-3076-4976-8eba-9cc42b321dbc" containerID="c1ea50be29fa2ff0898b95ed1aae929c26a917baf44ad2cbf7b25a2b5a4995d7" exitCode=0 Oct 30 00:39:55.596730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:55.596415 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" event={"ID":"3c119ce3-3076-4976-8eba-9cc42b321dbc","Type":"ContainerDied","Data":"c1ea50be29fa2ff0898b95ed1aae929c26a917baf44ad2cbf7b25a2b5a4995d7"} Oct 30 00:39:55.596730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:55.596718 4905 scope.go:122] "RemoveContainer" containerID="c1ea50be29fa2ff0898b95ed1aae929c26a917baf44ad2cbf7b25a2b5a4995d7" Oct 30 00:39:56.602146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:39:56.602115 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-skktb" event={"ID":"3c119ce3-3076-4976-8eba-9cc42b321dbc","Type":"ContainerStarted","Data":"bc7f615478f7dcf5531da68e7c251730d28f2c3059e2309f317b089a1a0f889d"} Oct 30 00:40:10.650952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:10.650920 4905 generic.go:356] "Generic (PLEG): container finished" podID="86bb8832-40cc-49f3-b7f2-247e47c55c18" containerID="a44b5c74fb28bf50d6bcd45af47f497b24d72e6cfebeb26a93ec0caac4689cc5" exitCode=0 Oct 30 00:40:10.651336 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:10.650974 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zws9n" event={"ID":"86bb8832-40cc-49f3-b7f2-247e47c55c18","Type":"ContainerDied","Data":"a44b5c74fb28bf50d6bcd45af47f497b24d72e6cfebeb26a93ec0caac4689cc5"} Oct 30 00:40:10.651336 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:10.651235 4905 scope.go:122] "RemoveContainer" containerID="a44b5c74fb28bf50d6bcd45af47f497b24d72e6cfebeb26a93ec0caac4689cc5" Oct 30 00:40:11.340842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.340811 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-857zf-wxsth"] Oct 30 00:40:11.341185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.341168 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9f2b8428-edb2-419a-9766-c05a126a03ab" containerName="registry" Oct 30 00:40:11.341272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.341188 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2b8428-edb2-419a-9766-c05a126a03ab" containerName="registry" Oct 30 00:40:11.341272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.341205 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d2739e90-51e9-431f-911d-2b3bfa587d04" containerName="registry" Oct 30 00:40:11.341272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.341212 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2739e90-51e9-431f-911d-2b3bfa587d04" containerName="registry" Oct 30 00:40:11.341419 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.341289 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2b8428-edb2-419a-9766-c05a126a03ab" containerName="registry" Oct 30 00:40:11.341419 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.341302 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2739e90-51e9-431f-911d-2b3bfa587d04" containerName="registry" Oct 30 00:40:11.344417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.344394 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.370057 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.370035 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-857zf-wxsth"] Oct 30 00:40:11.511070 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.511039 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-service-ca-bundle\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.511170 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.511078 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vng\" (UniqueName: \"kubernetes.io/projected/de030612-5f15-4bf9-a630-5d541f87279e-kube-api-access-96vng\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.511170 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.511103 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de030612-5f15-4bf9-a630-5d541f87279e-archives-path\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.511237 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.511223 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-trusted-ca-bundle\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.612034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.611965 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-service-ca-bundle\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.612034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.612007 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vng\" (UniqueName: \"kubernetes.io/projected/de030612-5f15-4bf9-a630-5d541f87279e-kube-api-access-96vng\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.612212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.612035 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de030612-5f15-4bf9-a630-5d541f87279e-archives-path\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.612212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.612073 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-trusted-ca-bundle\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.612457 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.612433 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de030612-5f15-4bf9-a630-5d541f87279e-archives-path\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.612712 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.612694 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-service-ca-bundle\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.612836 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.612817 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-trusted-ca-bundle\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.634039 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.634018 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vng\" (UniqueName: \"kubernetes.io/projected/de030612-5f15-4bf9-a630-5d541f87279e-kube-api-access-96vng\") pod \"periodic-gathering-857zf-wxsth\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.655039 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.655022 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:40:11.656981 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.656962 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zws9n" event={"ID":"86bb8832-40cc-49f3-b7f2-247e47c55c18","Type":"ContainerStarted","Data":"8b18668eff2f2543c5e8e8cfc2ea051b5df9b7f40a290682f0be22ba470c0689"} Oct 30 00:40:11.945202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:11.945181 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-857zf-wxsth"] Oct 30 00:40:11.947007 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:40:11.946982 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde030612_5f15_4bf9_a630_5d541f87279e.slice/crio-44c5e219c676b6ea3c9b9e47f79a8390deae200e56da66bd6b6c971b1736bf96 WatchSource:0}: Error finding container 44c5e219c676b6ea3c9b9e47f79a8390deae200e56da66bd6b6c971b1736bf96: Status 404 returned error can't find the container with id 44c5e219c676b6ea3c9b9e47f79a8390deae200e56da66bd6b6c971b1736bf96 Oct 30 00:40:12.412853 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:40:12.412818 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:40:12.662566 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:12.662513 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-857zf-wxsth" event={"ID":"de030612-5f15-4bf9-a630-5d541f87279e","Type":"ContainerStarted","Data":"d64a0cb7b42c86f4dc2681e4e1cb797f794373300c99d089bd6dbe3daccb22d8"} Oct 30 00:40:12.662991 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:12.662576 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-857zf-wxsth" event={"ID":"de030612-5f15-4bf9-a630-5d541f87279e","Type":"ContainerStarted","Data":"44c5e219c676b6ea3c9b9e47f79a8390deae200e56da66bd6b6c971b1736bf96"} Oct 30 00:40:12.707416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:40:12.707369 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-857zf-wxsth" podStartSLOduration=1.707353331 podStartE2EDuration="1.707353331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:12.707080831 +0000 UTC m=+97.650254023" watchObservedRunningTime="2026-10-30 00:40:12.707353331 +0000 UTC m=+97.650526527" Oct 30 00:40:42.411555 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:40:42.411503 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:41:12.412823 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:41:12.412741 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:41:35.946603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:35.946568 4905 generic.go:356] "Generic (PLEG): container finished" podID="de030612-5f15-4bf9-a630-5d541f87279e" containerID="d64a0cb7b42c86f4dc2681e4e1cb797f794373300c99d089bd6dbe3daccb22d8" exitCode=0 Oct 30 00:41:35.946994 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:35.946642 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-857zf-wxsth" event={"ID":"de030612-5f15-4bf9-a630-5d541f87279e","Type":"ContainerDied","Data":"d64a0cb7b42c86f4dc2681e4e1cb797f794373300c99d089bd6dbe3daccb22d8"} Oct 30 00:41:37.071559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.071536 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:41:37.183317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.183286 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de030612-5f15-4bf9-a630-5d541f87279e-kube-api-access-96vng\" (UniqueName: \"kubernetes.io/projected/de030612-5f15-4bf9-a630-5d541f87279e-kube-api-access-96vng\") pod \"de030612-5f15-4bf9-a630-5d541f87279e\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " Oct 30 00:41:37.183496 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.183329 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-service-ca-bundle\") pod \"de030612-5f15-4bf9-a630-5d541f87279e\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " Oct 30 00:41:37.183496 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.183362 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de030612-5f15-4bf9-a630-5d541f87279e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de030612-5f15-4bf9-a630-5d541f87279e-archives-path\") pod \"de030612-5f15-4bf9-a630-5d541f87279e\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " Oct 30 00:41:37.183496 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.183392 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-trusted-ca-bundle\") pod \"de030612-5f15-4bf9-a630-5d541f87279e\" (UID: \"de030612-5f15-4bf9-a630-5d541f87279e\") " Oct 30 00:41:37.183745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.183716 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-service-ca-bundle" pod "de030612-5f15-4bf9-a630-5d541f87279e" (UID: "de030612-5f15-4bf9-a630-5d541f87279e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:41:37.183797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.183764 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de030612-5f15-4bf9-a630-5d541f87279e-archives-path" pod "de030612-5f15-4bf9-a630-5d541f87279e" (UID: "de030612-5f15-4bf9-a630-5d541f87279e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:37.183797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.183785 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-trusted-ca-bundle" pod "de030612-5f15-4bf9-a630-5d541f87279e" (UID: "de030612-5f15-4bf9-a630-5d541f87279e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:41:37.185722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.185690 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de030612-5f15-4bf9-a630-5d541f87279e-kube-api-access-96vng" pod "de030612-5f15-4bf9-a630-5d541f87279e" (UID: "de030612-5f15-4bf9-a630-5d541f87279e"). InnerVolumeSpecName "kube-api-access-96vng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:37.284565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.284530 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vng\" (UniqueName: \"kubernetes.io/projected/de030612-5f15-4bf9-a630-5d541f87279e-kube-api-access-96vng\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:41:37.284565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.284557 4905 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-service-ca-bundle\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:41:37.284744 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.284573 4905 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de030612-5f15-4bf9-a630-5d541f87279e-archives-path\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:41:37.284744 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.284587 4905 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de030612-5f15-4bf9-a630-5d541f87279e-trusted-ca-bundle\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:41:37.954390 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.954344 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-857zf-wxsth" event={"ID":"de030612-5f15-4bf9-a630-5d541f87279e","Type":"ContainerDied","Data":"44c5e219c676b6ea3c9b9e47f79a8390deae200e56da66bd6b6c971b1736bf96"} Oct 30 00:41:37.954390 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.954394 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44c5e219c676b6ea3c9b9e47f79a8390deae200e56da66bd6b6c971b1736bf96" Oct 30 00:41:37.954619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:41:37.954362 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-857zf-wxsth" Oct 30 00:41:42.412363 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:41:42.412328 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:42:12.411781 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:42:12.411748 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:42:42.412013 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:42:42.411974 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:43:12.411534 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:43:12.411485 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:43:17.875070 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.875041 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm"] Oct 30 00:43:17.875472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.875316 4905 cpu_manager.go:414] "containerMap: removing container" podUID="de030612-5f15-4bf9-a630-5d541f87279e" containerName="insights-gathering" Oct 30 00:43:17.875472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.875326 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de030612-5f15-4bf9-a630-5d541f87279e" containerName="insights-gathering" Oct 30 00:43:17.875472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.875386 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="de030612-5f15-4bf9-a630-5d541f87279e" containerName="insights-gathering" Oct 30 00:43:17.878058 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.878043 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:17.910545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.910510 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hj56n\"" Oct 30 00:43:17.910545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.910534 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 00:43:17.910695 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.910589 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 00:43:17.911462 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:17.911446 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm"] Oct 30 00:43:18.055626 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.055600 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/199f2072-d9ae-4009-9502-c6c035977c44-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fjtbm\" (UID: \"199f2072-d9ae-4009-9502-c6c035977c44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:18.055742 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.055631 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pffj\" (UniqueName: \"kubernetes.io/projected/199f2072-d9ae-4009-9502-c6c035977c44-kube-api-access-2pffj\") pod \"cert-manager-webhook-5dbbbf75cc-fjtbm\" (UID: \"199f2072-d9ae-4009-9502-c6c035977c44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:18.156797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.156744 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/199f2072-d9ae-4009-9502-c6c035977c44-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fjtbm\" (UID: \"199f2072-d9ae-4009-9502-c6c035977c44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:18.156797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.156777 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pffj\" (UniqueName: \"kubernetes.io/projected/199f2072-d9ae-4009-9502-c6c035977c44-kube-api-access-2pffj\") pod \"cert-manager-webhook-5dbbbf75cc-fjtbm\" (UID: \"199f2072-d9ae-4009-9502-c6c035977c44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:18.181559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.181531 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/199f2072-d9ae-4009-9502-c6c035977c44-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fjtbm\" (UID: \"199f2072-d9ae-4009-9502-c6c035977c44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:18.181824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.181805 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pffj\" (UniqueName: \"kubernetes.io/projected/199f2072-d9ae-4009-9502-c6c035977c44-kube-api-access-2pffj\") pod \"cert-manager-webhook-5dbbbf75cc-fjtbm\" (UID: \"199f2072-d9ae-4009-9502-c6c035977c44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:18.198611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.198582 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:18.491184 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:18.491161 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm"] Oct 30 00:43:18.492455 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:43:18.492427 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod199f2072_d9ae_4009_9502_c6c035977c44.slice/crio-c700b6f0b8d21568b6b37479d3a133cce4d43de03d897abf5c9f481302a2f552 WatchSource:0}: Error finding container c700b6f0b8d21568b6b37479d3a133cce4d43de03d897abf5c9f481302a2f552: Status 404 returned error can't find the container with id c700b6f0b8d21568b6b37479d3a133cce4d43de03d897abf5c9f481302a2f552 Oct 30 00:43:19.283127 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:19.283088 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" event={"ID":"199f2072-d9ae-4009-9502-c6c035977c44","Type":"ContainerStarted","Data":"c700b6f0b8d21568b6b37479d3a133cce4d43de03d897abf5c9f481302a2f552"} Oct 30 00:43:22.296575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:22.296540 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" event={"ID":"199f2072-d9ae-4009-9502-c6c035977c44","Type":"ContainerStarted","Data":"43e44f9c8de5719b7734342612c0e9cf1c78f3e047e2910016087de34d8bcec1"} Oct 30 00:43:22.296925 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:22.296628 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:22.343827 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:22.343786 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" podStartSLOduration=2.464642406 podStartE2EDuration="5.34377563s" totalImagesPullingTime="2.879133224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:22.343643452 +0000 UTC m=+287.286816648" watchObservedRunningTime="2026-10-30 00:43:22.34377563 +0000 UTC m=+287.286948816" Oct 30 00:43:28.306075 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:28.306052 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" Oct 30 00:43:35.196784 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:35.196757 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:43:35.198838 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:35.198816 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:43:37.271218 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.271189 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q"] Oct 30 00:43:37.278025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.278002 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.293556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.293511 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee7f9e1a-0987-4812-bfba-97cad9512ab1-tmp\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.293896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.293619 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.293896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.293684 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-metrics-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.293896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.293724 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee7f9e1a-0987-4812-bfba-97cad9512ab1-manifests\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.293896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.293768 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-webhook-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.293896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.293837 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzxd\" (UniqueName: \"kubernetes.io/projected/ee7f9e1a-0987-4812-bfba-97cad9512ab1-kube-api-access-jfzxd\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.295398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.295377 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:43:37.295570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.295502 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:43:37.295846 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.295833 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:43:37.295887 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.295834 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:43:37.309320 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.309287 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f4n97\"" Oct 30 00:43:37.309320 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.309293 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:43:37.321133 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.321111 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q"] Oct 30 00:43:37.394186 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.394162 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.394314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.394197 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-metrics-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.394314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.394215 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee7f9e1a-0987-4812-bfba-97cad9512ab1-manifests\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.394314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.394238 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-webhook-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.394476 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.394370 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzxd\" (UniqueName: \"kubernetes.io/projected/ee7f9e1a-0987-4812-bfba-97cad9512ab1-kube-api-access-jfzxd\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.394476 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.394445 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee7f9e1a-0987-4812-bfba-97cad9512ab1-tmp\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.394662 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.394644 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee7f9e1a-0987-4812-bfba-97cad9512ab1-manifests\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.396580 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.396559 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee7f9e1a-0987-4812-bfba-97cad9512ab1-tmp\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.396718 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.396700 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-webhook-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.396930 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.396912 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.396965 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.396912 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee7f9e1a-0987-4812-bfba-97cad9512ab1-metrics-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.418494 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.418474 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzxd\" (UniqueName: \"kubernetes.io/projected/ee7f9e1a-0987-4812-bfba-97cad9512ab1-kube-api-access-jfzxd\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-wzp4q\" (UID: \"ee7f9e1a-0987-4812-bfba-97cad9512ab1\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.593297 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.593239 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:37.928113 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:37.928084 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q"] Oct 30 00:43:37.928782 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:43:37.928762 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7f9e1a_0987_4812_bfba_97cad9512ab1.slice/crio-223b6f631c953fc4956a49ddc273ac321409f8b3ceca21ab9d0d6e0bc3dc4f79 WatchSource:0}: Error finding container 223b6f631c953fc4956a49ddc273ac321409f8b3ceca21ab9d0d6e0bc3dc4f79: Status 404 returned error can't find the container with id 223b6f631c953fc4956a49ddc273ac321409f8b3ceca21ab9d0d6e0bc3dc4f79 Oct 30 00:43:38.359453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:38.359424 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" event={"ID":"ee7f9e1a-0987-4812-bfba-97cad9512ab1","Type":"ContainerStarted","Data":"223b6f631c953fc4956a49ddc273ac321409f8b3ceca21ab9d0d6e0bc3dc4f79"} Oct 30 00:43:41.378393 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:41.378364 4905 generic.go:356] "Generic (PLEG): container finished" podID="ee7f9e1a-0987-4812-bfba-97cad9512ab1" containerID="9989390b778ee3c24b77073c695a47e1aca8a340b5d25f27b9e9b4be80ea9dee" exitCode=0 Oct 30 00:43:41.378806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:41.378453 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" event={"ID":"ee7f9e1a-0987-4812-bfba-97cad9512ab1","Type":"ContainerDied","Data":"9989390b778ee3c24b77073c695a47e1aca8a340b5d25f27b9e9b4be80ea9dee"} Oct 30 00:43:41.378926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:41.378912 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:43:42.384693 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:42.384655 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" event={"ID":"ee7f9e1a-0987-4812-bfba-97cad9512ab1","Type":"ContainerStarted","Data":"207213a687b31bab1180473c7af7da6cf395a8c77a2f6f578328f941fa531e20"} Oct 30 00:43:42.385081 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:42.384741 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:42.412594 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:43:42.412560 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:43:42.481887 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:42.481837 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" podStartSLOduration=1.844528795 podStartE2EDuration="5.481822633s" totalImagesPullingTime="3.167838769s" totalInitContainerRuntime="469.455069ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:42.46743829 +0000 UTC m=+307.410611516" watchObservedRunningTime="2026-10-30 00:43:42.481822633 +0000 UTC m=+307.424995818" Oct 30 00:43:53.389964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:53.389888 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q" Oct 30 00:43:57.082808 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.082776 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g"] Oct 30 00:43:57.087927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.087905 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.118915 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.118889 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 00:43:57.119078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.118942 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 00:43:57.119078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.118990 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-s98tb\"" Oct 30 00:43:57.120317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.120282 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g"] Oct 30 00:43:57.126062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.126041 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-tmp\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.126157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.126072 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-tls-certs\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.126240 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.126223 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkmp9\" (UniqueName: \"kubernetes.io/projected/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-kube-api-access-nkmp9\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.226847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.226823 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkmp9\" (UniqueName: \"kubernetes.io/projected/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-kube-api-access-nkmp9\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.226987 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.226869 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-tmp\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.226987 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.226901 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-tls-certs\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.229355 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.229328 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-tmp\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.229470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.229449 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-tls-certs\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.247188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.247164 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkmp9\" (UniqueName: \"kubernetes.io/projected/f9305cf4-d3b1-43cc-b248-33a8d2bbca5c-kube-api-access-nkmp9\") pod \"kube-auth-proxy-7597b65cbf-jb59g\" (UID: \"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c\") " pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.400363 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.400299 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" Oct 30 00:43:57.691098 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:57.691074 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g"] Oct 30 00:43:57.692254 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:43:57.692234 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9305cf4_d3b1_43cc_b248_33a8d2bbca5c.slice/crio-9133b4a87d53c510b971a96e2ef1fb6f97befcc1bbefc4819ad4d0e859d4e188 WatchSource:0}: Error finding container 9133b4a87d53c510b971a96e2ef1fb6f97befcc1bbefc4819ad4d0e859d4e188: Status 404 returned error can't find the container with id 9133b4a87d53c510b971a96e2ef1fb6f97befcc1bbefc4819ad4d0e859d4e188 Oct 30 00:43:58.338996 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.338967 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt"] Oct 30 00:43:58.342486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.342454 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.363062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.363025 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:43:58.363499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.363433 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 00:43:58.363499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.363462 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 00:43:58.363499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.363495 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 00:43:58.377049 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.376731 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lrqvm\"" Oct 30 00:43:58.378587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.378568 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt"] Oct 30 00:43:58.437070 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.437040 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwt7m\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-kube-api-access-kwt7m\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.437190 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.437098 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.437190 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.437143 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b688193-b1f1-4869-8e5f-3f98525e8b42-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.437307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.437184 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.437307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.437221 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.437307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.437259 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.437307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.437294 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.449559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.449467 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" event={"ID":"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c","Type":"ContainerStarted","Data":"9133b4a87d53c510b971a96e2ef1fb6f97befcc1bbefc4819ad4d0e859d4e188"} Oct 30 00:43:58.538001 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.537963 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwt7m\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-kube-api-access-kwt7m\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.538141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.538006 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.538141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.538032 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b688193-b1f1-4869-8e5f-3f98525e8b42-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.538141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.538056 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.538141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.538072 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.538141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.538100 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.538141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.538132 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.539416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.539392 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.541770 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.541701 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.541770 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.541704 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b688193-b1f1-4869-8e5f-3f98525e8b42-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.542303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.542278 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.542902 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.542878 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.561106 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.561087 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.562281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.562262 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwt7m\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-kube-api-access-kwt7m\") pod \"istiod-openshift-gateway-7876bfcb74-sxrlt\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.661133 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.661057 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:43:58.979383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:58.979353 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt"] Oct 30 00:43:58.980352 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:43:58.980325 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b688193_b1f1_4869_8e5f_3f98525e8b42.slice/crio-fb2c5e1128ca222cbd4504a8a30b22ea06152382f7f196207f8aeb6675c8789e WatchSource:0}: Error finding container fb2c5e1128ca222cbd4504a8a30b22ea06152382f7f196207f8aeb6675c8789e: Status 404 returned error can't find the container with id fb2c5e1128ca222cbd4504a8a30b22ea06152382f7f196207f8aeb6675c8789e Oct 30 00:43:59.456193 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:43:59.456156 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" event={"ID":"2b688193-b1f1-4869-8e5f-3f98525e8b42","Type":"ContainerStarted","Data":"fb2c5e1128ca222cbd4504a8a30b22ea06152382f7f196207f8aeb6675c8789e"} Oct 30 00:44:00.461124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:00.461046 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" event={"ID":"f9305cf4-d3b1-43cc-b248-33a8d2bbca5c","Type":"ContainerStarted","Data":"e2c7e91cf571812be87c60882fd39d9f6fc385643b184b1be095b65cde5b7984"} Oct 30 00:44:00.503373 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:00.503328 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7597b65cbf-jb59g" podStartSLOduration=1.157043939 podStartE2EDuration="3.503314734s" totalImagesPullingTime="2.346270795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:00.502183594 +0000 UTC m=+325.445356782" watchObservedRunningTime="2026-10-30 00:44:00.503314734 +0000 UTC m=+325.446487926" Oct 30 00:44:12.412681 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:44:12.412638 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:44:16.232257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:16.232226 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:44:16.232505 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:16.232293 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:44:16.526061 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:16.526026 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" event={"ID":"2b688193-b1f1-4869-8e5f-3f98525e8b42","Type":"ContainerStarted","Data":"152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175"} Oct 30 00:44:16.526230 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:16.526164 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:44:16.583768 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:16.583726 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" podStartSLOduration=1.334622831 podStartE2EDuration="18.583715856s" totalImagesPullingTime="17.249093025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:16.583114 +0000 UTC m=+341.526287192" watchObservedRunningTime="2026-10-30 00:44:16.583715856 +0000 UTC m=+341.526889048" Oct 30 00:44:17.531814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.531781 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:44:17.562640 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.562614 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj"] Oct 30 00:44:17.566016 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.565997 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.578790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.578758 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj"] Oct 30 00:44:17.582029 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.582008 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qlf2v\"" Oct 30 00:44:17.689417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689385 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689416 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689451 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689591 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689625 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689660 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689698 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689715 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq54c\" (UniqueName: \"kubernetes.io/projected/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-kube-api-access-mq54c\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.689848 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.689785 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.790677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790613 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.790677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790648 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.790677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790669 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790713 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790744 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790773 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790820 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790850 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.790873 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq54c\" (UniqueName: \"kubernetes.io/projected/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-kube-api-access-mq54c\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.791372 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.791592 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.791671 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.791734 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.791937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.791827 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.793098 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.793076 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.794143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.794124 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.810922 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.810898 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.811260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.811242 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq54c\" (UniqueName: \"kubernetes.io/projected/9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a-kube-api-access-mq54c\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d54frhj\" (UID: \"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:17.879737 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:17.879713 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:18.185115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:18.185082 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj"] Oct 30 00:44:18.186239 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:44:18.186218 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9e4cb9_5297_40fb_aa9c_ec8bf348bc6a.slice/crio-54afbe531a179a42779f2845aa51490b35f0f2ba60cc25434d8162f89fd63cff WatchSource:0}: Error finding container 54afbe531a179a42779f2845aa51490b35f0f2ba60cc25434d8162f89fd63cff: Status 404 returned error can't find the container with id 54afbe531a179a42779f2845aa51490b35f0f2ba60cc25434d8162f89fd63cff Oct 30 00:44:18.534444 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:18.534416 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" event={"ID":"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a","Type":"ContainerStarted","Data":"54afbe531a179a42779f2845aa51490b35f0f2ba60cc25434d8162f89fd63cff"} Oct 30 00:44:21.388283 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:21.388253 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:44:21.388565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:21.388323 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:44:21.388565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:21.388353 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:44:21.549676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:21.549646 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" event={"ID":"9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a","Type":"ContainerStarted","Data":"53d3d4365d33b8c90d5af1c44a8a52c73a2134510b157a2ec78f9df0c44a45cd"} Oct 30 00:44:21.880609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:21.880585 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:22.884278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:22.884253 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:22.955438 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:22.955389 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" podStartSLOduration=2.756306421 podStartE2EDuration="5.955376931s" totalImagesPullingTime="3.19907051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:21.621686881 +0000 UTC m=+346.564860073" watchObservedRunningTime="2026-10-30 00:44:22.955376931 +0000 UTC m=+347.898550123" Oct 30 00:44:23.557579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:23.557550 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:23.558384 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:23.558368 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d54frhj" Oct 30 00:44:42.412777 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:44:42.412746 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:44:57.620224 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.620193 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k"] Oct 30 00:44:57.631953 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.631926 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.648626 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.648597 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k"] Oct 30 00:44:57.784725 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.784699 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.784725 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.784727 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.784898 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.784750 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e2681e-21d4-42a6-967c-d462361c99e6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.784898 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.784764 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.784898 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.784814 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e2681e-21d4-42a6-967c-d462361c99e6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.784991 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.784917 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e2681e-21d4-42a6-967c-d462361c99e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.784991 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.784953 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-225bn\" (UniqueName: \"kubernetes.io/projected/26e2681e-21d4-42a6-967c-d462361c99e6-kube-api-access-225bn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.886296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.886211 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.886296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.886246 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.886296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.886281 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e2681e-21d4-42a6-967c-d462361c99e6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.886513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.886304 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.886513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.886327 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e2681e-21d4-42a6-967c-d462361c99e6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.886513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.886392 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e2681e-21d4-42a6-967c-d462361c99e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.886513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.886437 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-225bn\" (UniqueName: \"kubernetes.io/projected/26e2681e-21d4-42a6-967c-d462361c99e6-kube-api-access-225bn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.887307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.887212 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e2681e-21d4-42a6-967c-d462361c99e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.888834 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.888812 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e2681e-21d4-42a6-967c-d462361c99e6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.888947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.888883 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.888947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.888889 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.888947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.888924 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e2681e-21d4-42a6-967c-d462361c99e6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.909502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.909480 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e2681e-21d4-42a6-967c-d462361c99e6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.910002 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.909983 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-225bn\" (UniqueName: \"kubernetes.io/projected/26e2681e-21d4-42a6-967c-d462361c99e6-kube-api-access-225bn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rf4k\" (UID: \"26e2681e-21d4-42a6-967c-d462361c99e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:57.942808 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:57.942792 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:58.247569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:58.247540 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k"] Oct 30 00:44:58.248300 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:44:58.248277 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e2681e_21d4_42a6_967c_d462361c99e6.slice/crio-e02b2840fd4dbcd3b74043277f3dd03d9f926d18c82ed946c437d501d85ff970 WatchSource:0}: Error finding container e02b2840fd4dbcd3b74043277f3dd03d9f926d18c82ed946c437d501d85ff970: Status 404 returned error can't find the container with id e02b2840fd4dbcd3b74043277f3dd03d9f926d18c82ed946c437d501d85ff970 Oct 30 00:44:58.250345 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:58.250311 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:44:58.250408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:58.250396 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:44:58.699333 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:58.699257 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" event={"ID":"26e2681e-21d4-42a6-967c-d462361c99e6","Type":"ContainerStarted","Data":"bb63b402b86fecb71be63d436394c605427629208e9c6f5c79eaaca0efc7b81d"} Oct 30 00:44:58.699333 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:58.699305 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" event={"ID":"26e2681e-21d4-42a6-967c-d462361c99e6","Type":"ContainerStarted","Data":"e02b2840fd4dbcd3b74043277f3dd03d9f926d18c82ed946c437d501d85ff970"} Oct 30 00:44:58.699333 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:58.699334 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:58.775071 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:58.775015 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" podStartSLOduration=1.7749961779999999 podStartE2EDuration="1.774996178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:58.76026387 +0000 UTC m=+383.703437064" watchObservedRunningTime="2026-10-30 00:44:58.774996178 +0000 UTC m=+383.718169372" Oct 30 00:44:59.704377 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:59.704348 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rf4k" Oct 30 00:44:59.775056 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:59.775027 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt"] Oct 30 00:44:59.775219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:44:59.775199 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" podUID="2b688193-b1f1-4869-8e5f-3f98525e8b42" containerName="discovery" containerID="cri-o://152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175" gracePeriod=30 Oct 30 00:45:00.016863 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.016843 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:45:00.105167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105145 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-kubeconfig\") pod \"2b688193-b1f1-4869-8e5f-3f98525e8b42\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " Oct 30 00:45:00.105311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105177 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-token\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-token\") pod \"2b688193-b1f1-4869-8e5f-3f98525e8b42\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " Oct 30 00:45:00.105311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105220 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b688193-b1f1-4869-8e5f-3f98525e8b42-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b688193-b1f1-4869-8e5f-3f98525e8b42-local-certs\") pod \"2b688193-b1f1-4869-8e5f-3f98525e8b42\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " Oct 30 00:45:00.105311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105237 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-cacerts\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-cacerts\") pod \"2b688193-b1f1-4869-8e5f-3f98525e8b42\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " Oct 30 00:45:00.105311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105278 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-ca-configmap\") pod \"2b688193-b1f1-4869-8e5f-3f98525e8b42\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " Oct 30 00:45:00.105311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105303 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-kube-api-access-kwt7m\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-kube-api-access-kwt7m\") pod \"2b688193-b1f1-4869-8e5f-3f98525e8b42\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " Oct 30 00:45:00.105600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105323 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-dns-cert\") pod \"2b688193-b1f1-4869-8e5f-3f98525e8b42\" (UID: \"2b688193-b1f1-4869-8e5f-3f98525e8b42\") " Oct 30 00:45:00.105749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.105720 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-ca-configmap" pod "2b688193-b1f1-4869-8e5f-3f98525e8b42" (UID: "2b688193-b1f1-4869-8e5f-3f98525e8b42"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:00.107624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.107600 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-kubeconfig" pod "2b688193-b1f1-4869-8e5f-3f98525e8b42" (UID: "2b688193-b1f1-4869-8e5f-3f98525e8b42"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:00.107790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.107762 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-dns-cert" pod "2b688193-b1f1-4869-8e5f-3f98525e8b42" (UID: "2b688193-b1f1-4869-8e5f-3f98525e8b42"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:00.108011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.107984 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-cacerts" pod "2b688193-b1f1-4869-8e5f-3f98525e8b42" (UID: "2b688193-b1f1-4869-8e5f-3f98525e8b42"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:00.108011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.108001 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b688193-b1f1-4869-8e5f-3f98525e8b42-local-certs" pod "2b688193-b1f1-4869-8e5f-3f98525e8b42" (UID: "2b688193-b1f1-4869-8e5f-3f98525e8b42"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:45:00.108133 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.108037 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-kube-api-access-kwt7m" pod "2b688193-b1f1-4869-8e5f-3f98525e8b42" (UID: "2b688193-b1f1-4869-8e5f-3f98525e8b42"). InnerVolumeSpecName "kube-api-access-kwt7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:00.108133 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.108091 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-token" pod "2b688193-b1f1-4869-8e5f-3f98525e8b42" (UID: "2b688193-b1f1-4869-8e5f-3f98525e8b42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:00.206202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.206176 4905 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-kubeconfig\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:45:00.206329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.206206 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:45:00.206329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.206220 4905 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b688193-b1f1-4869-8e5f-3f98525e8b42-local-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:45:00.206329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.206232 4905 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-cacerts\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:45:00.206329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.206245 4905 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-ca-configmap\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:45:00.206329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.206258 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwt7m\" (UniqueName: \"kubernetes.io/projected/2b688193-b1f1-4869-8e5f-3f98525e8b42-kube-api-access-kwt7m\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:45:00.206329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.206278 4905 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b688193-b1f1-4869-8e5f-3f98525e8b42-istio-csr-dns-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:45:00.707695 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.707659 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b688193-b1f1-4869-8e5f-3f98525e8b42" containerID="152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175" exitCode=0 Oct 30 00:45:00.708126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.707705 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" Oct 30 00:45:00.708126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.707749 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" event={"ID":"2b688193-b1f1-4869-8e5f-3f98525e8b42","Type":"ContainerDied","Data":"152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175"} Oct 30 00:45:00.708126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.707804 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt" event={"ID":"2b688193-b1f1-4869-8e5f-3f98525e8b42","Type":"ContainerDied","Data":"fb2c5e1128ca222cbd4504a8a30b22ea06152382f7f196207f8aeb6675c8789e"} Oct 30 00:45:00.708126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.707833 4905 scope.go:122] "RemoveContainer" containerID="152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175" Oct 30 00:45:00.716207 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.716188 4905 scope.go:122] "RemoveContainer" containerID="152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175" Oct 30 00:45:00.716464 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:45:00.716444 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175\": container with ID starting with 152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175 not found: ID does not exist" containerID="152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175" Oct 30 00:45:00.716534 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.716472 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175"} err="failed to get container status \"152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175\": rpc error: code = NotFound desc = could not find container \"152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175\": container with ID starting with 152241abfc87b5eae924fd698c04011df7c9726aaa660ddf9b63affe5c82e175 not found: ID does not exist" Oct 30 00:45:00.777100 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.777076 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt"] Oct 30 00:45:00.780513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:00.780496 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sxrlt"] Oct 30 00:45:01.188055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:45:01.188031 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b688193-b1f1-4869-8e5f-3f98525e8b42" path="/var/lib/kubelet/pods/2b688193-b1f1-4869-8e5f-3f98525e8b42/volumes" Oct 30 00:45:12.412224 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:45:12.412183 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:45:42.413071 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:45:42.413030 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:46:07.375939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.375908 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p"] Oct 30 00:46:07.376402 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.376217 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2b688193-b1f1-4869-8e5f-3f98525e8b42" containerName="discovery" Oct 30 00:46:07.376402 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.376227 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b688193-b1f1-4869-8e5f-3f98525e8b42" containerName="discovery" Oct 30 00:46:07.376402 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.376289 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b688193-b1f1-4869-8e5f-3f98525e8b42" containerName="discovery" Oct 30 00:46:07.379209 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.379192 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:07.392546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.392495 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p"] Oct 30 00:46:07.405076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.405046 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 00:46:07.405076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.405046 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 00:46:07.405241 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.405098 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cld2j\"" Oct 30 00:46:07.473800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.473779 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:07.473907 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.473806 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltz6r\" (UniqueName: \"kubernetes.io/projected/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-kube-api-access-ltz6r\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:07.574855 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.574832 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:07.574947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.574866 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltz6r\" (UniqueName: \"kubernetes.io/projected/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-kube-api-access-ltz6r\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:07.575165 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.575148 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:07.596756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.596735 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltz6r\" (UniqueName: \"kubernetes.io/projected/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-kube-api-access-ltz6r\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:07.692106 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:07.692058 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:08.029233 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:08.029197 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p"] Oct 30 00:46:08.040560 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:46:08.040535 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9e8d1d_0e95_4ab5_931e_04131b4a79ac.slice/crio-a86ca30388b594f3bfadc901681db7044d1cd72f4a0cc55d0edce5a77cd62372 WatchSource:0}: Error finding container a86ca30388b594f3bfadc901681db7044d1cd72f4a0cc55d0edce5a77cd62372: Status 404 returned error can't find the container with id a86ca30388b594f3bfadc901681db7044d1cd72f4a0cc55d0edce5a77cd62372 Oct 30 00:46:08.977760 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:08.977712 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" event={"ID":"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac","Type":"ContainerStarted","Data":"a86ca30388b594f3bfadc901681db7044d1cd72f4a0cc55d0edce5a77cd62372"} Oct 30 00:46:09.306785 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.306752 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b"] Oct 30 00:46:09.310235 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.310212 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:09.340632 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.340591 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-75f28\"" Oct 30 00:46:09.342191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.342168 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b"] Oct 30 00:46:09.389719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.389697 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbt5h\" (UniqueName: \"kubernetes.io/projected/c070d479-e0b1-42c3-85e6-6152ec9c4158-kube-api-access-jbt5h\") pod \"limitador-operator-controller-manager-856dd584c5-w2f9b\" (UID: \"c070d479-e0b1-42c3-85e6-6152ec9c4158\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:09.490884 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.490855 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbt5h\" (UniqueName: \"kubernetes.io/projected/c070d479-e0b1-42c3-85e6-6152ec9c4158-kube-api-access-jbt5h\") pod \"limitador-operator-controller-manager-856dd584c5-w2f9b\" (UID: \"c070d479-e0b1-42c3-85e6-6152ec9c4158\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:09.514820 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.514793 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbt5h\" (UniqueName: \"kubernetes.io/projected/c070d479-e0b1-42c3-85e6-6152ec9c4158-kube-api-access-jbt5h\") pod \"limitador-operator-controller-manager-856dd584c5-w2f9b\" (UID: \"c070d479-e0b1-42c3-85e6-6152ec9c4158\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:09.626398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.626332 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:09.930389 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.930362 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b"] Oct 30 00:46:09.986661 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:09.986628 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" event={"ID":"c070d479-e0b1-42c3-85e6-6152ec9c4158","Type":"ContainerStarted","Data":"c84893cf8d4e3b2736cf9149cb19ce13d0bbb333f88ef04d13b2a0cd10813bcd"} Oct 30 00:46:12.417718 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:12.417670 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:46:14.009662 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:14.009627 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" event={"ID":"c070d479-e0b1-42c3-85e6-6152ec9c4158","Type":"ContainerStarted","Data":"efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f"} Oct 30 00:46:14.010124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:14.009784 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:14.054019 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:14.053961 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" podStartSLOduration=1.286226819 podStartE2EDuration="5.053948483s" totalImagesPullingTime="3.767721664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:14.052635603 +0000 UTC m=+458.995808818" watchObservedRunningTime="2026-10-30 00:46:14.053948483 +0000 UTC m=+458.997121675" Oct 30 00:46:16.020082 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:16.020050 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" event={"ID":"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac","Type":"ContainerStarted","Data":"ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f"} Oct 30 00:46:16.020446 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:16.020123 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:16.093104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:16.093034 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" podStartSLOduration=2.074995677 podStartE2EDuration="9.093017277s" totalImagesPullingTime="7.0180216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:16.078114817 +0000 UTC m=+461.021288009" watchObservedRunningTime="2026-10-30 00:46:16.093017277 +0000 UTC m=+461.036190472" Oct 30 00:46:25.015993 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:25.015964 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:27.032306 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:27.032275 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:29.052749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.052720 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk"] Oct 30 00:46:29.056224 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.056201 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.073570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.073541 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk"] Oct 30 00:46:29.141295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.141266 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jp86\" (UniqueName: \"kubernetes.io/projected/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-kube-api-access-4jp86\") pod \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.141411 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.141330 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.241768 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.241746 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.241860 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.241783 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jp86\" (UniqueName: \"kubernetes.io/projected/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-kube-api-access-4jp86\") pod \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.242093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.242076 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.272110 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.272084 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jp86\" (UniqueName: \"kubernetes.io/projected/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-kube-api-access-4jp86\") pod \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.368119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.368064 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:29.679822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:29.679796 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk"] Oct 30 00:46:29.680219 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:46:29.680193 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6042a9ab_abe9_4ba3_ac3a_b9ef48ffc591.slice/crio-8a64bb86c4f28f78e3ae6977c33df220edb5fa34fedcd7b46fdf5ddba790b7da WatchSource:0}: Error finding container 8a64bb86c4f28f78e3ae6977c33df220edb5fa34fedcd7b46fdf5ddba790b7da: Status 404 returned error can't find the container with id 8a64bb86c4f28f78e3ae6977c33df220edb5fa34fedcd7b46fdf5ddba790b7da Oct 30 00:46:30.081926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.081890 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" event={"ID":"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591","Type":"ContainerStarted","Data":"fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b"} Oct 30 00:46:30.082445 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.081980 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" event={"ID":"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591","Type":"ContainerStarted","Data":"8a64bb86c4f28f78e3ae6977c33df220edb5fa34fedcd7b46fdf5ddba790b7da"} Oct 30 00:46:30.082445 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.082025 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:30.128966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.128900 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" podStartSLOduration=1.128882978 podStartE2EDuration="1.128882978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:30.127112622 +0000 UTC m=+475.070285815" watchObservedRunningTime="2026-10-30 00:46:30.128882978 +0000 UTC m=+475.072056170" Oct 30 00:46:30.702804 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.702773 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p"] Oct 30 00:46:30.702988 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.702956 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" containerName="manager" containerID="cri-o://ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f" gracePeriod=2 Oct 30 00:46:30.739566 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.739504 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p"] Oct 30 00:46:30.744993 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:30.744954 4905 status_manager.go:1068] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-6b6fb56846-2h28p\": pods \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:30.754145 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.754122 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk"] Oct 30 00:46:30.768585 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.768561 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk"] Oct 30 00:46:30.793571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.793540 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b"] Oct 30 00:46:30.793779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.793744 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" containerName="manager" containerID="cri-o://efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f" gracePeriod=2 Oct 30 00:46:30.800073 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.800052 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r"] Oct 30 00:46:30.800378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.800366 4905 cpu_manager.go:414] "containerMap: removing container" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" containerName="manager" Oct 30 00:46:30.800417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.800380 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" containerName="manager" Oct 30 00:46:30.800417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.800393 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" containerName="manager" Oct 30 00:46:30.800417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.800397 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" containerName="manager" Oct 30 00:46:30.800511 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.800457 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" containerName="manager" Oct 30 00:46:30.800511 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.800465 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" containerName="manager" Oct 30 00:46:30.803824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.803807 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:30.811412 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:30.811378 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:30.815860 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.815840 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b"] Oct 30 00:46:30.826557 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:30.826483 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w2f9b\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:30.830877 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.830857 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r"] Oct 30 00:46:30.840365 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.840344 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh"] Oct 30 00:46:30.840665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.840653 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" containerName="manager" Oct 30 00:46:30.840715 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.840666 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" containerName="manager" Oct 30 00:46:30.840749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.840738 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" containerName="manager" Oct 30 00:46:30.843563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.843546 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" Oct 30 00:46:30.873868 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.873846 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh"] Oct 30 00:46:30.903029 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:30.902991 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:30.918034 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:30.918006 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w2f9b\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:30.933215 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:30.933177 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:30.948051 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:30.948023 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w2f9b\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:30.955984 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.955919 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dnzt\" (UniqueName: \"kubernetes.io/projected/d95e6b8b-edad-4f7f-bd7b-f796430174e8-kube-api-access-9dnzt\") pod \"kuadrant-operator-controller-manager-956956bc4-2t94r\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:30.955984 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.955958 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95e6b8b-edad-4f7f-bd7b-f796430174e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-2t94r\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:30.956107 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:30.956080 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4ss\" (UniqueName: \"kubernetes.io/projected/7b06bf07-f94a-4f87-af61-69f35a22aeb2-kube-api-access-fb4ss\") pod \"limitador-operator-controller-manager-856dd584c5-nh5jh\" (UID: \"7b06bf07-f94a-4f87-af61-69f35a22aeb2\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" Oct 30 00:46:31.036084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.036065 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:31.038761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.038744 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:31.051233 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.051207 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:31.056640 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.056618 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4ss\" (UniqueName: \"kubernetes.io/projected/7b06bf07-f94a-4f87-af61-69f35a22aeb2-kube-api-access-fb4ss\") pod \"limitador-operator-controller-manager-856dd584c5-nh5jh\" (UID: \"7b06bf07-f94a-4f87-af61-69f35a22aeb2\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" Oct 30 00:46:31.056718 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.056674 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dnzt\" (UniqueName: \"kubernetes.io/projected/d95e6b8b-edad-4f7f-bd7b-f796430174e8-kube-api-access-9dnzt\") pod \"kuadrant-operator-controller-manager-956956bc4-2t94r\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:31.056718 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.056701 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95e6b8b-edad-4f7f-bd7b-f796430174e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-2t94r\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:31.057001 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.056987 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95e6b8b-edad-4f7f-bd7b-f796430174e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-2t94r\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:31.066058 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.066035 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w2f9b\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:31.081024 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.080994 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:31.081150 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.081133 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4ss\" (UniqueName: \"kubernetes.io/projected/7b06bf07-f94a-4f87-af61-69f35a22aeb2-kube-api-access-fb4ss\") pod \"limitador-operator-controller-manager-856dd584c5-nh5jh\" (UID: \"7b06bf07-f94a-4f87-af61-69f35a22aeb2\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" Oct 30 00:46:31.081755 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.081732 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dnzt\" (UniqueName: \"kubernetes.io/projected/d95e6b8b-edad-4f7f-bd7b-f796430174e8-kube-api-access-9dnzt\") pod \"kuadrant-operator-controller-manager-956956bc4-2t94r\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:31.087480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.087457 4905 generic.go:356] "Generic (PLEG): container finished" podID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" containerID="ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f" exitCode=0 Oct 30 00:46:31.087770 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.087497 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:31.087770 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.087534 4905 scope.go:122] "RemoveContainer" containerID="ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f" Oct 30 00:46:31.088909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.088889 4905 generic.go:356] "Generic (PLEG): container finished" podID="c070d479-e0b1-42c3-85e6-6152ec9c4158" containerID="efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f" exitCode=0 Oct 30 00:46:31.089001 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.088935 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:31.095316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.095300 4905 scope.go:122] "RemoveContainer" containerID="ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f" Oct 30 00:46:31.095602 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.095566 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f\": container with ID starting with ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f not found: ID does not exist" containerID="ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f" Oct 30 00:46:31.095649 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.095614 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f"} err="failed to get container status \"ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f\": rpc error: code = NotFound desc = could not find container \"ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f\": container with ID starting with ba33d6fb1498686c1641a5c02b6a98e7a1ec423fab6850b0f0c2b3f5521ce66f not found: ID does not exist" Oct 30 00:46:31.095649 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.095631 4905 scope.go:122] "RemoveContainer" containerID="efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f" Oct 30 00:46:31.095938 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.095910 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w2f9b\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:31.102749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.102731 4905 scope.go:122] "RemoveContainer" containerID="efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f" Oct 30 00:46:31.103010 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.102992 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f\": container with ID starting with efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f not found: ID does not exist" containerID="efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f" Oct 30 00:46:31.103078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.103014 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f"} err="failed to get container status \"efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f\": rpc error: code = NotFound desc = could not find container \"efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f\": container with ID starting with efbe00c1c89dd8ea3e15b42dfcefe46a415887ab25273d3ccb2220ef4844309f not found: ID does not exist" Oct 30 00:46:31.110983 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.110962 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2h28p\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2h28p" Oct 30 00:46:31.126008 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:31.125988 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w2f9b\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w2f9b" Oct 30 00:46:31.157424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.157403 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c070d479-e0b1-42c3-85e6-6152ec9c4158-kube-api-access-jbt5h\" (UniqueName: \"kubernetes.io/projected/c070d479-e0b1-42c3-85e6-6152ec9c4158-kube-api-access-jbt5h\") pod \"c070d479-e0b1-42c3-85e6-6152ec9c4158\" (UID: \"c070d479-e0b1-42c3-85e6-6152ec9c4158\") " Oct 30 00:46:31.157501 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.157458 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-kube-api-access-ltz6r\" (UniqueName: \"kubernetes.io/projected/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-kube-api-access-ltz6r\") pod \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " Oct 30 00:46:31.157501 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.157486 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-extensions-socket-volume\") pod \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\" (UID: \"cb9e8d1d-0e95-4ab5-931e-04131b4a79ac\") " Oct 30 00:46:31.158011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.157951 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-extensions-socket-volume" pod "cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" (UID: "cb9e8d1d-0e95-4ab5-931e-04131b4a79ac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:46:31.159483 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.159458 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c070d479-e0b1-42c3-85e6-6152ec9c4158-kube-api-access-jbt5h" pod "c070d479-e0b1-42c3-85e6-6152ec9c4158" (UID: "c070d479-e0b1-42c3-85e6-6152ec9c4158"). InnerVolumeSpecName "kube-api-access-jbt5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:46:31.159577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.159509 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-kube-api-access-ltz6r" pod "cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" (UID: "cb9e8d1d-0e95-4ab5-931e-04131b4a79ac"). InnerVolumeSpecName "kube-api-access-ltz6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:46:31.175201 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.175184 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:31.181127 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.181112 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" Oct 30 00:46:31.192402 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.192377 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c070d479-e0b1-42c3-85e6-6152ec9c4158" path="/var/lib/kubelet/pods/c070d479-e0b1-42c3-85e6-6152ec9c4158/volumes" Oct 30 00:46:31.192740 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.192726 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9e8d1d-0e95-4ab5-931e-04131b4a79ac" path="/var/lib/kubelet/pods/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac/volumes" Oct 30 00:46:31.258947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.258919 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltz6r\" (UniqueName: \"kubernetes.io/projected/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-kube-api-access-ltz6r\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:46:31.259037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.258949 4905 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9e8d1d-0e95-4ab5-931e-04131b4a79ac-extensions-socket-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:46:31.259037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.258962 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbt5h\" (UniqueName: \"kubernetes.io/projected/c070d479-e0b1-42c3-85e6-6152ec9c4158-kube-api-access-jbt5h\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:46:31.498593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.498564 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh"] Oct 30 00:46:31.499143 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:46:31.499119 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b06bf07_f94a_4f87_af61_69f35a22aeb2.slice/crio-858feaac15431fe14c71cb06619d07087a5ab2186e9afcaa0bf07eb815440356 WatchSource:0}: Error finding container 858feaac15431fe14c71cb06619d07087a5ab2186e9afcaa0bf07eb815440356: Status 404 returned error can't find the container with id 858feaac15431fe14c71cb06619d07087a5ab2186e9afcaa0bf07eb815440356 Oct 30 00:46:31.502924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:31.502906 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r"] Oct 30 00:46:31.510125 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:46:31.510104 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95e6b8b_edad_4f7f_bd7b_f796430174e8.slice/crio-49e2d96da67a3ed3309fdaf6994072a9c65f2712abd2fd93093a798bfae96883 WatchSource:0}: Error finding container 49e2d96da67a3ed3309fdaf6994072a9c65f2712abd2fd93093a798bfae96883: Status 404 returned error can't find the container with id 49e2d96da67a3ed3309fdaf6994072a9c65f2712abd2fd93093a798bfae96883 Oct 30 00:46:32.094937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.094715 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" event={"ID":"d95e6b8b-edad-4f7f-bd7b-f796430174e8","Type":"ContainerStarted","Data":"525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977"} Oct 30 00:46:32.094937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.094768 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" event={"ID":"d95e6b8b-edad-4f7f-bd7b-f796430174e8","Type":"ContainerStarted","Data":"49e2d96da67a3ed3309fdaf6994072a9c65f2712abd2fd93093a798bfae96883"} Oct 30 00:46:32.094937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.094874 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:32.097191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.097167 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" event={"ID":"7b06bf07-f94a-4f87-af61-69f35a22aeb2","Type":"ContainerStarted","Data":"9097138c7b403b1090d8b8b9f2ef75b69e898b1aba378e646c951932f139d745"} Oct 30 00:46:32.097324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.097208 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" Oct 30 00:46:32.097324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.097223 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" event={"ID":"7b06bf07-f94a-4f87-af61-69f35a22aeb2","Type":"ContainerStarted","Data":"858feaac15431fe14c71cb06619d07087a5ab2186e9afcaa0bf07eb815440356"} Oct 30 00:46:32.098207 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.098167 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" podUID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" containerName="manager" containerID="cri-o://fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b" gracePeriod=2 Oct 30 00:46:32.168913 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.168873 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" podStartSLOduration=2.168859358 podStartE2EDuration="2.168859358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:32.154105864 +0000 UTC m=+477.097279055" watchObservedRunningTime="2026-10-30 00:46:32.168859358 +0000 UTC m=+477.112032546" Oct 30 00:46:32.198071 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.198019 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" podStartSLOduration=2.198007969 podStartE2EDuration="2.198007969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:32.196876566 +0000 UTC m=+477.140049757" watchObservedRunningTime="2026-10-30 00:46:32.198007969 +0000 UTC m=+477.141181161" Oct 30 00:46:32.355404 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.355353 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:32.468244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.468218 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-extensions-socket-volume\") pod \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " Oct 30 00:46:32.468380 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.468339 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-kube-api-access-4jp86\" (UniqueName: \"kubernetes.io/projected/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-kube-api-access-4jp86\") pod \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\" (UID: \"6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591\") " Oct 30 00:46:32.468658 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.468627 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-extensions-socket-volume" pod "6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" (UID: "6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:46:32.470432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.470407 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-kube-api-access-4jp86" pod "6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" (UID: "6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591"). InnerVolumeSpecName "kube-api-access-4jp86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:46:32.569749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.569716 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jp86\" (UniqueName: \"kubernetes.io/projected/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-kube-api-access-4jp86\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:46:32.569749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:32.569744 4905 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591-extensions-socket-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:46:33.103811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:33.103781 4905 generic.go:356] "Generic (PLEG): container finished" podID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" containerID="fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b" exitCode=0 Oct 30 00:46:33.104260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:33.103826 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:33.104260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:33.103888 4905 scope.go:122] "RemoveContainer" containerID="fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b" Oct 30 00:46:33.112244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:33.112228 4905 scope.go:122] "RemoveContainer" containerID="fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b" Oct 30 00:46:33.112488 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:33.112470 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b\": container with ID starting with fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b not found: ID does not exist" containerID="fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b" Oct 30 00:46:33.112665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:33.112496 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b"} err="failed to get container status \"fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b\": rpc error: code = NotFound desc = could not find container \"fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b\": container with ID starting with fad5ee746792a27cbb8dc6b3187ad5fc2506a4f0f34a537664744fb65a10dd8b not found: ID does not exist" Oct 30 00:46:33.121065 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:33.121031 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:33.135894 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:33.135874 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-xbdmk\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-xbdmk" Oct 30 00:46:33.188750 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:33.188729 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591" path="/var/lib/kubelet/pods/6042a9ab-abe9-4ba3-ac3a-b9ef48ffc591/volumes" Oct 30 00:46:42.412331 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:42.412294 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:46:43.106177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.106151 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nh5jh" Oct 30 00:46:43.106343 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.106232 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:43.447375 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.447307 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48"] Oct 30 00:46:43.455305 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.455278 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:43.479200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.479177 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48"] Oct 30 00:46:43.546205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.546182 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4xll\" (UniqueName: \"kubernetes.io/projected/7ef33b08-200a-4979-ac2e-856e3c89db4c-kube-api-access-t4xll\") pod \"kuadrant-operator-controller-manager-57bb9bfff-drc48\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:43.546328 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.546238 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef33b08-200a-4979-ac2e-856e3c89db4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-drc48\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:43.646937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.646915 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4xll\" (UniqueName: \"kubernetes.io/projected/7ef33b08-200a-4979-ac2e-856e3c89db4c-kube-api-access-t4xll\") pod \"kuadrant-operator-controller-manager-57bb9bfff-drc48\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:43.647045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.646967 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef33b08-200a-4979-ac2e-856e3c89db4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-drc48\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:43.647259 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.647244 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef33b08-200a-4979-ac2e-856e3c89db4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-drc48\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:43.667836 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.667815 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4xll\" (UniqueName: \"kubernetes.io/projected/7ef33b08-200a-4979-ac2e-856e3c89db4c-kube-api-access-t4xll\") pod \"kuadrant-operator-controller-manager-57bb9bfff-drc48\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:43.767414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:43.767391 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:44.081384 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:44.081320 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48"] Oct 30 00:46:44.093032 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:46:44.093007 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef33b08_200a_4979_ac2e_856e3c89db4c.slice/crio-e8175c3746b3c90a624d0863d697ad57c45e0fb501f0d7ebefe810c11497bf9b WatchSource:0}: Error finding container e8175c3746b3c90a624d0863d697ad57c45e0fb501f0d7ebefe810c11497bf9b: Status 404 returned error can't find the container with id e8175c3746b3c90a624d0863d697ad57c45e0fb501f0d7ebefe810c11497bf9b Oct 30 00:46:44.150766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:44.150738 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" event={"ID":"7ef33b08-200a-4979-ac2e-856e3c89db4c","Type":"ContainerStarted","Data":"e8175c3746b3c90a624d0863d697ad57c45e0fb501f0d7ebefe810c11497bf9b"} Oct 30 00:46:45.157777 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:45.157701 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" event={"ID":"7ef33b08-200a-4979-ac2e-856e3c89db4c","Type":"ContainerStarted","Data":"79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e"} Oct 30 00:46:45.158164 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:45.157822 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:45.222205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:45.222163 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" podStartSLOduration=2.222153117 podStartE2EDuration="2.222153117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:45.220961483 +0000 UTC m=+490.164134674" watchObservedRunningTime="2026-10-30 00:46:45.222153117 +0000 UTC m=+490.165326308" Oct 30 00:46:56.164827 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.164800 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 00:46:56.318397 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.318362 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r"] Oct 30 00:46:56.318632 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.318611 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" podUID="d95e6b8b-edad-4f7f-bd7b-f796430174e8" containerName="manager" containerID="cri-o://525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977" gracePeriod=10 Oct 30 00:46:56.555044 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.555021 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:56.637419 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.637391 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95e6b8b-edad-4f7f-bd7b-f796430174e8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95e6b8b-edad-4f7f-bd7b-f796430174e8-extensions-socket-volume\") pod \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " Oct 30 00:46:56.637558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.637542 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95e6b8b-edad-4f7f-bd7b-f796430174e8-kube-api-access-9dnzt\" (UniqueName: \"kubernetes.io/projected/d95e6b8b-edad-4f7f-bd7b-f796430174e8-kube-api-access-9dnzt\") pod \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\" (UID: \"d95e6b8b-edad-4f7f-bd7b-f796430174e8\") " Oct 30 00:46:56.637766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.637746 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95e6b8b-edad-4f7f-bd7b-f796430174e8-extensions-socket-volume" pod "d95e6b8b-edad-4f7f-bd7b-f796430174e8" (UID: "d95e6b8b-edad-4f7f-bd7b-f796430174e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:46:56.639486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.639464 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95e6b8b-edad-4f7f-bd7b-f796430174e8-kube-api-access-9dnzt" pod "d95e6b8b-edad-4f7f-bd7b-f796430174e8" (UID: "d95e6b8b-edad-4f7f-bd7b-f796430174e8"). InnerVolumeSpecName "kube-api-access-9dnzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:46:56.738571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.738549 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dnzt\" (UniqueName: \"kubernetes.io/projected/d95e6b8b-edad-4f7f-bd7b-f796430174e8-kube-api-access-9dnzt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:46:56.738571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:56.738572 4905 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95e6b8b-edad-4f7f-bd7b-f796430174e8-extensions-socket-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:46:57.207165 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.207139 4905 generic.go:356] "Generic (PLEG): container finished" podID="d95e6b8b-edad-4f7f-bd7b-f796430174e8" containerID="525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977" exitCode=0 Oct 30 00:46:57.207473 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.207180 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" Oct 30 00:46:57.207473 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.207220 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" event={"ID":"d95e6b8b-edad-4f7f-bd7b-f796430174e8","Type":"ContainerDied","Data":"525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977"} Oct 30 00:46:57.207473 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.207263 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r" event={"ID":"d95e6b8b-edad-4f7f-bd7b-f796430174e8","Type":"ContainerDied","Data":"49e2d96da67a3ed3309fdaf6994072a9c65f2712abd2fd93093a798bfae96883"} Oct 30 00:46:57.207473 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.207286 4905 scope.go:122] "RemoveContainer" containerID="525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977" Oct 30 00:46:57.215122 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.215110 4905 scope.go:122] "RemoveContainer" containerID="525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977" Oct 30 00:46:57.215389 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:46:57.215372 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977\": container with ID starting with 525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977 not found: ID does not exist" containerID="525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977" Oct 30 00:46:57.215447 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.215401 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977"} err="failed to get container status \"525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977\": rpc error: code = NotFound desc = could not find container \"525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977\": container with ID starting with 525e34e7539f0c56a8164930ccfccbb82ae1387e728f18c02904934e49022977 not found: ID does not exist" Oct 30 00:46:57.261982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.261960 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r"] Oct 30 00:46:57.275757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:57.275736 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2t94r"] Oct 30 00:46:59.187445 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:46:59.187413 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95e6b8b-edad-4f7f-bd7b-f796430174e8" path="/var/lib/kubelet/pods/d95e6b8b-edad-4f7f-bd7b-f796430174e8/volumes" Oct 30 00:47:00.337931 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.337898 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c"] Oct 30 00:47:00.338479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.338394 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d95e6b8b-edad-4f7f-bd7b-f796430174e8" containerName="manager" Oct 30 00:47:00.338479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.338413 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95e6b8b-edad-4f7f-bd7b-f796430174e8" containerName="manager" Oct 30 00:47:00.338601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.338546 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95e6b8b-edad-4f7f-bd7b-f796430174e8" containerName="manager" Oct 30 00:47:00.344245 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.344225 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.364681 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364654 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.364681 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364682 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.364830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364712 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.364830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364785 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.364830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364816 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.364937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364843 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.364937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364861 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4p76\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-kube-api-access-g4p76\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.365052 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.364951 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.365052 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.365033 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.366387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.366370 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rxg2r\"" Oct 30 00:47:00.378350 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.378327 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c"] Oct 30 00:47:00.466234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466196 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466237 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4p76\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-kube-api-access-g4p76\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466259 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466286 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466350 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466379 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466489 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466555 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466573 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466792 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466839 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.466909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.466889 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.467060 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.467001 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.467271 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.467249 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.468957 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.468935 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.469136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.469117 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.488270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.488243 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.488381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.488364 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4p76\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-kube-api-access-g4p76\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6w6c\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.659065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.658985 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:00.963702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.963675 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c"] Oct 30 00:47:00.964074 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:47:00.964048 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774bdea2_f6dc_4d88_9cc9_91af7d21ae12.slice/crio-f1a4f91522d36238290ddf378890eea3b7295f5eb8ce93e36430e9b7b653dc9b WatchSource:0}: Error finding container f1a4f91522d36238290ddf378890eea3b7295f5eb8ce93e36430e9b7b653dc9b: Status 404 returned error can't find the container with id f1a4f91522d36238290ddf378890eea3b7295f5eb8ce93e36430e9b7b653dc9b Oct 30 00:47:00.966174 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.966143 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:47:00.966246 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.966209 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:47:00.966288 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:00.966251 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:47:01.226615 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:01.226507 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" event={"ID":"774bdea2-f6dc-4d88-9cc9-91af7d21ae12","Type":"ContainerStarted","Data":"683c692b6c7bf3f54b3992c1c282885468c7f6f5b59c6b08bed03dfcbdb75efa"} Oct 30 00:47:01.226615 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:01.226601 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" event={"ID":"774bdea2-f6dc-4d88-9cc9-91af7d21ae12","Type":"ContainerStarted","Data":"f1a4f91522d36238290ddf378890eea3b7295f5eb8ce93e36430e9b7b653dc9b"} Oct 30 00:47:01.272961 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:01.272906 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" podStartSLOduration=1.272891129 podStartE2EDuration="1.272891129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:01.27223734 +0000 UTC m=+506.215410513" watchObservedRunningTime="2026-10-30 00:47:01.272891129 +0000 UTC m=+506.216064329" Oct 30 00:47:01.659231 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:01.659200 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:02.664399 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:02.664373 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:03.235604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:03.235568 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:03.236413 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:03.236396 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:47:12.411649 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:47:12.411617 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:47:16.354845 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.354816 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tstr8"] Oct 30 00:47:16.359364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.359347 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.376587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.376565 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 00:47:16.376779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.376761 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fblw9\"" Oct 30 00:47:16.378006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.377980 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tstr8"] Oct 30 00:47:16.482485 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.482449 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tstr8"] Oct 30 00:47:16.492275 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.492249 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5274\" (UniqueName: \"kubernetes.io/projected/56ec76fb-4433-476d-ba36-637525f6d12e-kube-api-access-g5274\") pod \"limitador-limitador-74dfcfb894-tstr8\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.492412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.492383 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ec76fb-4433-476d-ba36-637525f6d12e-config-file\") pod \"limitador-limitador-74dfcfb894-tstr8\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.593048 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.593014 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ec76fb-4433-476d-ba36-637525f6d12e-config-file\") pod \"limitador-limitador-74dfcfb894-tstr8\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.593048 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.593050 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5274\" (UniqueName: \"kubernetes.io/projected/56ec76fb-4433-476d-ba36-637525f6d12e-kube-api-access-g5274\") pod \"limitador-limitador-74dfcfb894-tstr8\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.593754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.593729 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ec76fb-4433-476d-ba36-637525f6d12e-config-file\") pod \"limitador-limitador-74dfcfb894-tstr8\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.613534 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.613459 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5274\" (UniqueName: \"kubernetes.io/projected/56ec76fb-4433-476d-ba36-637525f6d12e-kube-api-access-g5274\") pod \"limitador-limitador-74dfcfb894-tstr8\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.671871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.671843 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:16.964308 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:16.964283 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tstr8"] Oct 30 00:47:16.965270 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:47:16.965244 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ec76fb_4433_476d_ba36_637525f6d12e.slice/crio-ef50f5f13346331272adb7580d1dea700a5d63ce5a291bc4cea43f315f2879a4 WatchSource:0}: Error finding container ef50f5f13346331272adb7580d1dea700a5d63ce5a291bc4cea43f315f2879a4: Status 404 returned error can't find the container with id ef50f5f13346331272adb7580d1dea700a5d63ce5a291bc4cea43f315f2879a4 Oct 30 00:47:17.291723 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:17.291692 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" event={"ID":"56ec76fb-4433-476d-ba36-637525f6d12e","Type":"ContainerStarted","Data":"ef50f5f13346331272adb7580d1dea700a5d63ce5a291bc4cea43f315f2879a4"} Oct 30 00:47:20.314372 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:20.314338 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" event={"ID":"56ec76fb-4433-476d-ba36-637525f6d12e","Type":"ContainerStarted","Data":"25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2"} Oct 30 00:47:20.314945 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:20.314592 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:20.355845 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:20.355802 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" podStartSLOduration=1.112554734 podStartE2EDuration="4.355791554s" totalImagesPullingTime="3.24323682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:20.355094418 +0000 UTC m=+525.298267609" watchObservedRunningTime="2026-10-30 00:47:20.355791554 +0000 UTC m=+525.298964743" Oct 30 00:47:31.232319 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.232286 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tstr8"] Oct 30 00:47:31.232711 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.232462 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" podUID="56ec76fb-4433-476d-ba36-637525f6d12e" containerName="limitador" containerID="cri-o://25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2" gracePeriod=30 Oct 30 00:47:31.233185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.233102 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:31.764492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.764465 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:31.917225 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.917107 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ec76fb-4433-476d-ba36-637525f6d12e-config-file\" (UniqueName: \"kubernetes.io/configmap/56ec76fb-4433-476d-ba36-637525f6d12e-config-file\") pod \"56ec76fb-4433-476d-ba36-637525f6d12e\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " Oct 30 00:47:31.917930 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.917690 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ec76fb-4433-476d-ba36-637525f6d12e-config-file" pod "56ec76fb-4433-476d-ba36-637525f6d12e" (UID: "56ec76fb-4433-476d-ba36-637525f6d12e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:47:31.918007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.917955 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ec76fb-4433-476d-ba36-637525f6d12e-kube-api-access-g5274\" (UniqueName: \"kubernetes.io/projected/56ec76fb-4433-476d-ba36-637525f6d12e-kube-api-access-g5274\") pod \"56ec76fb-4433-476d-ba36-637525f6d12e\" (UID: \"56ec76fb-4433-476d-ba36-637525f6d12e\") " Oct 30 00:47:31.918178 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.918154 4905 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ec76fb-4433-476d-ba36-637525f6d12e-config-file\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:47:31.920163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:31.920137 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ec76fb-4433-476d-ba36-637525f6d12e-kube-api-access-g5274" pod "56ec76fb-4433-476d-ba36-637525f6d12e" (UID: "56ec76fb-4433-476d-ba36-637525f6d12e"). InnerVolumeSpecName "kube-api-access-g5274". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:32.019277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.019254 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5274\" (UniqueName: \"kubernetes.io/projected/56ec76fb-4433-476d-ba36-637525f6d12e-kube-api-access-g5274\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:47:32.368177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.368143 4905 generic.go:356] "Generic (PLEG): container finished" podID="56ec76fb-4433-476d-ba36-637525f6d12e" containerID="25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2" exitCode=0 Oct 30 00:47:32.368584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.368185 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" event={"ID":"56ec76fb-4433-476d-ba36-637525f6d12e","Type":"ContainerDied","Data":"25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2"} Oct 30 00:47:32.368584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.368192 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" Oct 30 00:47:32.368584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.368233 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tstr8" event={"ID":"56ec76fb-4433-476d-ba36-637525f6d12e","Type":"ContainerDied","Data":"ef50f5f13346331272adb7580d1dea700a5d63ce5a291bc4cea43f315f2879a4"} Oct 30 00:47:32.368584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.368262 4905 scope.go:122] "RemoveContainer" containerID="25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2" Oct 30 00:47:32.376805 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.376790 4905 scope.go:122] "RemoveContainer" containerID="25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2" Oct 30 00:47:32.377037 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:47:32.377016 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2\": container with ID starting with 25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2 not found: ID does not exist" containerID="25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2" Oct 30 00:47:32.377100 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.377044 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2"} err="failed to get container status \"25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2\": rpc error: code = NotFound desc = could not find container \"25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2\": container with ID starting with 25cd8b732bce453788248efff45582e5d5070cfd6bf5543eb32e855cde74eec2 not found: ID does not exist" Oct 30 00:47:32.424600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.424543 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tstr8"] Oct 30 00:47:32.428077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:32.428058 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tstr8"] Oct 30 00:47:33.188773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:33.188740 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ec76fb-4433-476d-ba36-637525f6d12e" path="/var/lib/kubelet/pods/56ec76fb-4433-476d-ba36-637525f6d12e/volumes" Oct 30 00:47:42.412791 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:47:42.412755 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:47:48.383439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.383406 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ns5p8"] Oct 30 00:47:48.383926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.383742 4905 cpu_manager.go:414] "containerMap: removing container" podUID="56ec76fb-4433-476d-ba36-637525f6d12e" containerName="limitador" Oct 30 00:47:48.383926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.383753 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ec76fb-4433-476d-ba36-637525f6d12e" containerName="limitador" Oct 30 00:47:48.383926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.383808 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ec76fb-4433-476d-ba36-637525f6d12e" containerName="limitador" Oct 30 00:47:48.386447 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.386430 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:48.406028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.406006 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ns5p8"] Oct 30 00:47:48.418333 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.418311 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6b9qr\"" Oct 30 00:47:48.418441 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.418416 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 00:47:48.531204 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.531176 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ns5p8"] Oct 30 00:47:48.531393 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:47:48.531376 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8cmjd tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" podUID="a304ec30-8ab6-4ca9-bd7d-02e17930fcfe" Oct 30 00:47:48.553661 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.553630 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-tls-cert\") pod \"authorino-6889cbd8c4-ns5p8\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:48.553794 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.553670 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmjd\" (UniqueName: \"kubernetes.io/projected/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-kube-api-access-8cmjd\") pod \"authorino-6889cbd8c4-ns5p8\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:48.558556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.558508 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-747d44f99d-swksm"] Oct 30 00:47:48.561831 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.561814 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:48.581140 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.581121 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747d44f99d-swksm"] Oct 30 00:47:48.654840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.654779 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8904dc0e-a270-450b-8234-04ab21b9ed93-tls-cert\") pod \"authorino-747d44f99d-swksm\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:48.654840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.654819 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmgj\" (UniqueName: \"kubernetes.io/projected/8904dc0e-a270-450b-8234-04ab21b9ed93-kube-api-access-wgmgj\") pod \"authorino-747d44f99d-swksm\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:48.654994 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.654853 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-tls-cert\") pod \"authorino-6889cbd8c4-ns5p8\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:48.654994 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.654908 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmjd\" (UniqueName: \"kubernetes.io/projected/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-kube-api-access-8cmjd\") pod \"authorino-6889cbd8c4-ns5p8\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:48.657260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.657238 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-tls-cert\") pod \"authorino-6889cbd8c4-ns5p8\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:48.675343 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.675324 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmjd\" (UniqueName: \"kubernetes.io/projected/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-kube-api-access-8cmjd\") pod \"authorino-6889cbd8c4-ns5p8\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:48.755429 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.755405 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8904dc0e-a270-450b-8234-04ab21b9ed93-tls-cert\") pod \"authorino-747d44f99d-swksm\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:48.755546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.755443 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmgj\" (UniqueName: \"kubernetes.io/projected/8904dc0e-a270-450b-8234-04ab21b9ed93-kube-api-access-wgmgj\") pod \"authorino-747d44f99d-swksm\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:48.757716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.757690 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8904dc0e-a270-450b-8234-04ab21b9ed93-tls-cert\") pod \"authorino-747d44f99d-swksm\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:48.777740 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.777720 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmgj\" (UniqueName: \"kubernetes.io/projected/8904dc0e-a270-450b-8234-04ab21b9ed93-kube-api-access-wgmgj\") pod \"authorino-747d44f99d-swksm\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:48.872891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:48.872864 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:47:49.160634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.160602 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747d44f99d-swksm"] Oct 30 00:47:49.160977 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:47:49.160955 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8904dc0e_a270_450b_8234_04ab21b9ed93.slice/crio-f8dbb99406bde6a362f79148c1e472ba938890f6fd3e0390680cd3632c405657 WatchSource:0}: Error finding container f8dbb99406bde6a362f79148c1e472ba938890f6fd3e0390680cd3632c405657: Status 404 returned error can't find the container with id f8dbb99406bde6a362f79148c1e472ba938890f6fd3e0390680cd3632c405657 Oct 30 00:47:49.448662 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.448638 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:49.448662 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.448653 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d44f99d-swksm" event={"ID":"8904dc0e-a270-450b-8234-04ab21b9ed93","Type":"ContainerStarted","Data":"f8dbb99406bde6a362f79148c1e472ba938890f6fd3e0390680cd3632c405657"} Oct 30 00:47:49.456431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.456409 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:49.460561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.460545 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-kube-api-access-8cmjd\" (UniqueName: \"kubernetes.io/projected/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-kube-api-access-8cmjd\") pod \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " Oct 30 00:47:49.460636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.460592 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-tls-cert\" (UniqueName: \"kubernetes.io/secret/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-tls-cert\") pod \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\" (UID: \"a304ec30-8ab6-4ca9-bd7d-02e17930fcfe\") " Oct 30 00:47:49.462654 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.462633 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-kube-api-access-8cmjd" pod "a304ec30-8ab6-4ca9-bd7d-02e17930fcfe" (UID: "a304ec30-8ab6-4ca9-bd7d-02e17930fcfe"). InnerVolumeSpecName "kube-api-access-8cmjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:49.462654 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.462644 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-tls-cert" pod "a304ec30-8ab6-4ca9-bd7d-02e17930fcfe" (UID: "a304ec30-8ab6-4ca9-bd7d-02e17930fcfe"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:47:49.561340 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.561312 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cmjd\" (UniqueName: \"kubernetes.io/projected/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-kube-api-access-8cmjd\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:47:49.561340 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:49.561341 4905 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe-tls-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:47:50.454300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:50.454270 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ns5p8" Oct 30 00:47:50.531776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:50.531740 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ns5p8"] Oct 30 00:47:50.535995 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:50.535964 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ns5p8"] Oct 30 00:47:51.188556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:51.188508 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a304ec30-8ab6-4ca9-bd7d-02e17930fcfe" path="/var/lib/kubelet/pods/a304ec30-8ab6-4ca9-bd7d-02e17930fcfe/volumes" Oct 30 00:47:51.459665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:51.459587 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d44f99d-swksm" event={"ID":"8904dc0e-a270-450b-8234-04ab21b9ed93","Type":"ContainerStarted","Data":"3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f"} Oct 30 00:47:51.500000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:51.499951 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-747d44f99d-swksm" podStartSLOduration=1.45332335 podStartE2EDuration="3.499937638s" totalImagesPullingTime="2.046614288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:51.499057247 +0000 UTC m=+556.442230439" watchObservedRunningTime="2026-10-30 00:47:51.499937638 +0000 UTC m=+556.443110857" Oct 30 00:47:58.508762 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.508734 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-p422d"] Oct 30 00:47:58.512304 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.512287 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.526799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.526771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.526889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.526815 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4rq\" (UniqueName: \"kubernetes.io/projected/6422ed94-c582-4bb3-a559-2b774cf24b33-kube-api-access-sq4rq\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.526928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.526914 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-metrics-certs\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.528291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.528274 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 00:47:58.528386 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.528371 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 00:47:58.541777 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.541759 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w6xmx\"" Oct 30 00:47:58.541880 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.541830 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 00:47:58.543148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.543130 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-p422d"] Oct 30 00:47:58.627669 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.627640 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-metrics-certs\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.627827 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.627703 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.627827 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.627740 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4rq\" (UniqueName: \"kubernetes.io/projected/6422ed94-c582-4bb3-a559-2b774cf24b33-kube-api-access-sq4rq\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.627905 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:47:58.627845 4905 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Oct 30 00:47:58.627944 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:47:58.627923 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert podName:6422ed94-c582-4bb3-a559-2b774cf24b33 nodeName:}" failed. No retries permitted until 2026-10-30 00:47:59.127907294 +0000 UTC m=+564.071080468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert") pod "maas-controller-577f596567-p422d" (UID: "6422ed94-c582-4bb3-a559-2b774cf24b33") : secret "maas-controller-webhook-cert" not found Oct 30 00:47:58.630148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.630126 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-metrics-certs\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.648906 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.648882 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4rq\" (UniqueName: \"kubernetes.io/projected/6422ed94-c582-4bb3-a559-2b774cf24b33-kube-api-access-sq4rq\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:58.867428 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.867356 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-jn4xq"] Oct 30 00:47:58.870835 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.870820 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:58.892413 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.892379 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-jn4xq"] Oct 30 00:47:58.930279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.930251 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-metrics-certs\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:58.930475 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.930306 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzr2q\" (UniqueName: \"kubernetes.io/projected/5b38f224-a70d-4605-a35b-c4ae45ff122a-kube-api-access-tzr2q\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:58.930475 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:58.930407 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-webhook-cert\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.030968 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.030944 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-metrics-certs\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.031083 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.030994 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzr2q\" (UniqueName: \"kubernetes.io/projected/5b38f224-a70d-4605-a35b-c4ae45ff122a-kube-api-access-tzr2q\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.031129 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.031102 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-webhook-cert\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.033421 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.033398 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-metrics-certs\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.033534 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.033433 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-webhook-cert\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.051008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.050981 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzr2q\" (UniqueName: \"kubernetes.io/projected/5b38f224-a70d-4605-a35b-c4ae45ff122a-kube-api-access-tzr2q\") pod \"maas-controller-78d6f77c9d-jn4xq\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.132005 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.131948 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:59.134025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.134008 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert\") pod \"maas-controller-577f596567-p422d\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:59.183168 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.183145 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:47:59.424706 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.424627 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:47:59.474412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.473598 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-jn4xq"] Oct 30 00:47:59.476684 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:47:59.476440 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b38f224_a70d_4605_a35b_c4ae45ff122a.slice/crio-8f74f0009132fe6cc68473adc675bd04df1bc8c51850586a0ae519ee143e0285 WatchSource:0}: Error finding container 8f74f0009132fe6cc68473adc675bd04df1bc8c51850586a0ae519ee143e0285: Status 404 returned error can't find the container with id 8f74f0009132fe6cc68473adc675bd04df1bc8c51850586a0ae519ee143e0285 Oct 30 00:47:59.499951 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.499923 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" event={"ID":"5b38f224-a70d-4605-a35b-c4ae45ff122a","Type":"ContainerStarted","Data":"8f74f0009132fe6cc68473adc675bd04df1bc8c51850586a0ae519ee143e0285"} Oct 30 00:47:59.720475 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:47:59.720449 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-p422d"] Oct 30 00:47:59.721269 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:47:59.721242 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6422ed94_c582_4bb3_a559_2b774cf24b33.slice/crio-6a6010f86433c1edb1737b9d7b1fa67ad94c6256818ffc3c490ff40997788e0c WatchSource:0}: Error finding container 6a6010f86433c1edb1737b9d7b1fa67ad94c6256818ffc3c490ff40997788e0c: Status 404 returned error can't find the container with id 6a6010f86433c1edb1737b9d7b1fa67ad94c6256818ffc3c490ff40997788e0c Oct 30 00:48:00.508787 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:00.508749 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-p422d" event={"ID":"6422ed94-c582-4bb3-a559-2b774cf24b33","Type":"ContainerStarted","Data":"6a6010f86433c1edb1737b9d7b1fa67ad94c6256818ffc3c490ff40997788e0c"} Oct 30 00:48:02.522065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:02.522034 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-p422d" event={"ID":"6422ed94-c582-4bb3-a559-2b774cf24b33","Type":"ContainerStarted","Data":"f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02"} Oct 30 00:48:02.522503 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:02.522100 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:48:02.523686 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:02.523662 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" event={"ID":"5b38f224-a70d-4605-a35b-c4ae45ff122a","Type":"ContainerStarted","Data":"938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03"} Oct 30 00:48:02.523809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:02.523715 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:48:02.565866 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:02.565814 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-p422d" podStartSLOduration=1.9431875920000001 podStartE2EDuration="4.565800616s" totalImagesPullingTime="2.622613024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:02.564341764 +0000 UTC m=+567.507514956" watchObservedRunningTime="2026-10-30 00:48:02.565800616 +0000 UTC m=+567.508973807" Oct 30 00:48:02.610282 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:02.610232 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" podStartSLOduration=1.8911224880000002 podStartE2EDuration="4.610211163s" totalImagesPullingTime="2.719088675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:02.609044001 +0000 UTC m=+567.552217192" watchObservedRunningTime="2026-10-30 00:48:02.610211163 +0000 UTC m=+567.553384353" Oct 30 00:48:05.602188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.602151 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk"] Oct 30 00:48:05.605856 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.605834 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.626090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.626061 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk"] Oct 30 00:48:05.637544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.637506 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 00:48:05.637698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.637555 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 00:48:05.637698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.637610 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 00:48:05.637799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.637719 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-f5qff\"" Oct 30 00:48:05.681643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.681621 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cccc3e6c-ecee-4d0f-8719-1075e1789e73-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.681749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.681651 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cccc3e6c-ecee-4d0f-8719-1075e1789e73-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.681749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.681732 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccc3e6c-ecee-4d0f-8719-1075e1789e73-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.681840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.681764 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v6kd\" (UniqueName: \"kubernetes.io/projected/cccc3e6c-ecee-4d0f-8719-1075e1789e73-kube-api-access-2v6kd\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.782893 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.782867 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cccc3e6c-ecee-4d0f-8719-1075e1789e73-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.782893 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.782896 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cccc3e6c-ecee-4d0f-8719-1075e1789e73-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.783078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.782927 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccc3e6c-ecee-4d0f-8719-1075e1789e73-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.783078 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:48:05.783013 4905 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 00:48:05.783078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.783036 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6kd\" (UniqueName: \"kubernetes.io/projected/cccc3e6c-ecee-4d0f-8719-1075e1789e73-kube-api-access-2v6kd\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.783078 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:48:05.783069 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cccc3e6c-ecee-4d0f-8719-1075e1789e73-tls-cert podName:cccc3e6c-ecee-4d0f-8719-1075e1789e73 nodeName:}" failed. No retries permitted until 2026-10-30 00:48:06.28305381 +0000 UTC m=+571.226226981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/cccc3e6c-ecee-4d0f-8719-1075e1789e73-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-25kxk" (UID: "cccc3e6c-ecee-4d0f-8719-1075e1789e73") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 00:48:05.783704 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.783682 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccc3e6c-ecee-4d0f-8719-1075e1789e73-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.783834 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.783820 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cccc3e6c-ecee-4d0f-8719-1075e1789e73-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:05.803982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:05.803958 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v6kd\" (UniqueName: \"kubernetes.io/projected/cccc3e6c-ecee-4d0f-8719-1075e1789e73-kube-api-access-2v6kd\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:06.287100 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:06.287070 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cccc3e6c-ecee-4d0f-8719-1075e1789e73-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:06.289424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:06.289398 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cccc3e6c-ecee-4d0f-8719-1075e1789e73-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-25kxk\" (UID: \"cccc3e6c-ecee-4d0f-8719-1075e1789e73\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:06.518078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:06.518048 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:06.808949 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:06.808921 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk"] Oct 30 00:48:06.809390 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:48:06.809368 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccc3e6c_ecee_4d0f_8719_1075e1789e73.slice/crio-6bbc22db2379de2622f0d006fa169b6aff2946c961e1018c0a88a60508898629 WatchSource:0}: Error finding container 6bbc22db2379de2622f0d006fa169b6aff2946c961e1018c0a88a60508898629: Status 404 returned error can't find the container with id 6bbc22db2379de2622f0d006fa169b6aff2946c961e1018c0a88a60508898629 Oct 30 00:48:07.546733 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:07.546698 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" event={"ID":"cccc3e6c-ecee-4d0f-8719-1075e1789e73","Type":"ContainerStarted","Data":"6bbc22db2379de2622f0d006fa169b6aff2946c961e1018c0a88a60508898629"} Oct 30 00:48:12.417223 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:48:12.417099 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:48:13.531550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.531498 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 00:48:13.531924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.531617 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:48:13.648256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.648216 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-p422d"] Oct 30 00:48:13.648843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.648511 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-p422d" podUID="6422ed94-c582-4bb3-a559-2b774cf24b33" containerName="manager" containerID="cri-o://f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02" gracePeriod=30 Oct 30 00:48:13.924690 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.924661 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:48:13.957697 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.957665 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6422ed94-c582-4bb3-a559-2b774cf24b33-kube-api-access-sq4rq\" (UniqueName: \"kubernetes.io/projected/6422ed94-c582-4bb3-a559-2b774cf24b33-kube-api-access-sq4rq\") pod \"6422ed94-c582-4bb3-a559-2b774cf24b33\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " Oct 30 00:48:13.957846 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.957727 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert\") pod \"6422ed94-c582-4bb3-a559-2b774cf24b33\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " Oct 30 00:48:13.957846 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.957765 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-metrics-certs\") pod \"6422ed94-c582-4bb3-a559-2b774cf24b33\" (UID: \"6422ed94-c582-4bb3-a559-2b774cf24b33\") " Oct 30 00:48:13.960401 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.960352 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-metrics-certs" pod "6422ed94-c582-4bb3-a559-2b774cf24b33" (UID: "6422ed94-c582-4bb3-a559-2b774cf24b33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:13.960575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.960447 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert" pod "6422ed94-c582-4bb3-a559-2b774cf24b33" (UID: "6422ed94-c582-4bb3-a559-2b774cf24b33"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:13.960575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:13.960550 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6422ed94-c582-4bb3-a559-2b774cf24b33-kube-api-access-sq4rq" pod "6422ed94-c582-4bb3-a559-2b774cf24b33" (UID: "6422ed94-c582-4bb3-a559-2b774cf24b33"). InnerVolumeSpecName "kube-api-access-sq4rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:14.059189 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.059150 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq4rq\" (UniqueName: \"kubernetes.io/projected/6422ed94-c582-4bb3-a559-2b774cf24b33-kube-api-access-sq4rq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:48:14.059189 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.059190 4905 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-webhook-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:48:14.059385 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.059204 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6422ed94-c582-4bb3-a559-2b774cf24b33-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:48:14.587301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.587259 4905 generic.go:356] "Generic (PLEG): container finished" podID="6422ed94-c582-4bb3-a559-2b774cf24b33" containerID="f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02" exitCode=0 Oct 30 00:48:14.587772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.587305 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-p422d" Oct 30 00:48:14.587772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.587344 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-p422d" event={"ID":"6422ed94-c582-4bb3-a559-2b774cf24b33","Type":"ContainerDied","Data":"f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02"} Oct 30 00:48:14.587772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.587399 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-p422d" event={"ID":"6422ed94-c582-4bb3-a559-2b774cf24b33","Type":"ContainerDied","Data":"6a6010f86433c1edb1737b9d7b1fa67ad94c6256818ffc3c490ff40997788e0c"} Oct 30 00:48:14.587772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.587427 4905 scope.go:122] "RemoveContainer" containerID="f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02" Oct 30 00:48:14.597211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.597196 4905 scope.go:122] "RemoveContainer" containerID="f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02" Oct 30 00:48:14.597456 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:48:14.597440 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02\": container with ID starting with f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02 not found: ID does not exist" containerID="f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02" Oct 30 00:48:14.597547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.597464 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02"} err="failed to get container status \"f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02\": rpc error: code = NotFound desc = could not find container \"f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02\": container with ID starting with f94a66856583d0604d6c5c11797841437e18e5e825e168ccdbe5fb6e6d4d2a02 not found: ID does not exist" Oct 30 00:48:14.641703 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.641670 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-p422d"] Oct 30 00:48:14.655439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:14.655408 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-p422d"] Oct 30 00:48:15.190087 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:15.190002 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6422ed94-c582-4bb3-a559-2b774cf24b33" path="/var/lib/kubelet/pods/6422ed94-c582-4bb3-a559-2b774cf24b33/volumes" Oct 30 00:48:17.603103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:17.603064 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" event={"ID":"cccc3e6c-ecee-4d0f-8719-1075e1789e73","Type":"ContainerStarted","Data":"95e9ffc93d432bb99892064b7399c139f5db8e42da4c8e6275b7635c942f7ff5"} Oct 30 00:48:17.603506 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:17.603194 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:17.646394 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:17.646337 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" podStartSLOduration=2.810755055 podStartE2EDuration="12.646324246s" totalImagesPullingTime="9.835569191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:17.644780687 +0000 UTC m=+582.587953877" watchObservedRunningTime="2026-10-30 00:48:17.646324246 +0000 UTC m=+582.589497437" Oct 30 00:48:28.612927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:28.612902 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk" Oct 30 00:48:35.222835 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:35.222803 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:48:35.225078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:35.225058 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:48:42.412219 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:48:42.412185 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:48:47.739162 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:47.739086 4905 generic.go:356] "Generic (PLEG): container finished" podID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerID="683c692b6c7bf3f54b3992c1c282885468c7f6f5b59c6b08bed03dfcbdb75efa" exitCode=0 Oct 30 00:48:47.739567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:47.739163 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" event={"ID":"774bdea2-f6dc-4d88-9cc9-91af7d21ae12","Type":"ContainerDied","Data":"683c692b6c7bf3f54b3992c1c282885468c7f6f5b59c6b08bed03dfcbdb75efa"} Oct 30 00:48:47.739567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:47.739486 4905 scope.go:122] "RemoveContainer" containerID="683c692b6c7bf3f54b3992c1c282885468c7f6f5b59c6b08bed03dfcbdb75efa" Oct 30 00:48:47.740443 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:47.740419 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:48:47.740508 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:47.740483 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:48:47.740569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:47.740512 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:48:48.235981 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:48.235946 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:48:48.755502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:48.755469 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" event={"ID":"774bdea2-f6dc-4d88-9cc9-91af7d21ae12","Type":"ContainerStarted","Data":"6052c86a9256d846fbf72501069a36ef5073d9395b3cf168fb2bce94f575d9f0"} Oct 30 00:48:49.659389 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:49.659356 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:48:49.660595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:49.660569 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:48:49.660714 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:49.660614 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 30 00:48:50.660050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:50.660017 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:48:50.660439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:50.660069 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 30 00:48:51.660064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:51.660024 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:48:51.660582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:51.660082 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 30 00:48:52.660295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:52.660259 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6w6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:48:52.660709 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:52.660310 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 30 00:48:53.663911 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:53.663883 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:48:53.664332 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:53.664242 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:48:53.664961 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:48:53.664942 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 00:49:12.412769 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:49:12.412732 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:49:42.309405 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.309371 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g"] Oct 30 00:49:42.309780 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.309738 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6422ed94-c582-4bb3-a559-2b774cf24b33" containerName="manager" Oct 30 00:49:42.309780 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.309750 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6422ed94-c582-4bb3-a559-2b774cf24b33" containerName="manager" Oct 30 00:49:42.309856 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.309831 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6422ed94-c582-4bb3-a559-2b774cf24b33" containerName="manager" Oct 30 00:49:42.313049 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.313029 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.330926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.330905 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 00:49:42.344882 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.344860 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 00:49:42.345060 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.345002 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 00:49:42.346396 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.346372 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g"] Oct 30 00:49:42.382761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.382729 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.382869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.382767 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.382914 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.382868 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c45736-0323-495f-8968-2a96fe77d026-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.382914 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.382894 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5cn\" (UniqueName: \"kubernetes.io/projected/38c45736-0323-495f-8968-2a96fe77d026-kube-api-access-7d5cn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.383019 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.382922 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.383019 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.382945 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.413865 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:49:42.413841 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:49:42.430964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.430946 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8fnkv\"" Oct 30 00:49:42.483950 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.483924 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5cn\" (UniqueName: \"kubernetes.io/projected/38c45736-0323-495f-8968-2a96fe77d026-kube-api-access-7d5cn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484060 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.483967 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.484089 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.484164 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.484180 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.484220 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c45736-0323-495f-8968-2a96fe77d026-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484373 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.484354 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484514 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.484493 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.484514 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.484507 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.486316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.486298 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c45736-0323-495f-8968-2a96fe77d026-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.486589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.486573 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c45736-0323-495f-8968-2a96fe77d026-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.505302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.505276 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5cn\" (UniqueName: \"kubernetes.io/projected/38c45736-0323-495f-8968-2a96fe77d026-kube-api-access-7d5cn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g\" (UID: \"38c45736-0323-495f-8968-2a96fe77d026\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.626704 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.626648 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:42.922199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.922169 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g"] Oct 30 00:49:42.923404 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:49:42.923378 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c45736_0323_495f_8968_2a96fe77d026.slice/crio-b514aeaa5f7b8fd718f8d58e4e816263d6131e7abe588a060287268dfabd7b27 WatchSource:0}: Error finding container b514aeaa5f7b8fd718f8d58e4e816263d6131e7abe588a060287268dfabd7b27: Status 404 returned error can't find the container with id b514aeaa5f7b8fd718f8d58e4e816263d6131e7abe588a060287268dfabd7b27 Oct 30 00:49:42.924708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.924690 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:49:42.992240 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:42.992216 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" event={"ID":"38c45736-0323-495f-8968-2a96fe77d026","Type":"ContainerStarted","Data":"b514aeaa5f7b8fd718f8d58e4e816263d6131e7abe588a060287268dfabd7b27"} Oct 30 00:49:43.822991 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.822954 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2"] Oct 30 00:49:43.827176 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.827148 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:43.847321 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.847268 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 00:49:43.860136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.860107 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2"] Oct 30 00:49:43.899393 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.899349 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:43.899621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.899403 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:43.899621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.899465 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvprg\" (UniqueName: \"kubernetes.io/projected/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-kube-api-access-jvprg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:43.899621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.899485 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:43.899621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.899509 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:43.899621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:43.899563 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.000177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.000140 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.000357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.000235 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.000357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.000280 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.000554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.000365 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvprg\" (UniqueName: \"kubernetes.io/projected/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-kube-api-access-jvprg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.000554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.000391 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.000554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.000413 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.001284 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.001232 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.001499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.001478 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.001673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.001638 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.004005 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.003970 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.004227 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.004206 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.029964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.029906 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvprg\" (UniqueName: \"kubernetes.io/projected/20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec-kube-api-access-jvprg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2rzg2\" (UID: \"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.151709 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.151500 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:44.480568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:44.479393 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2"] Oct 30 00:49:44.485263 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:49:44.485230 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20db7a35_d107_4bda_a4ce_5f7d7a5ae6ec.slice/crio-2f48fa37a39d6a3b26c72226f878a69748dd46c0643518b86300cbc60fb85473 WatchSource:0}: Error finding container 2f48fa37a39d6a3b26c72226f878a69748dd46c0643518b86300cbc60fb85473: Status 404 returned error can't find the container with id 2f48fa37a39d6a3b26c72226f878a69748dd46c0643518b86300cbc60fb85473 Oct 30 00:49:45.005117 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:45.005002 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" event={"ID":"38c45736-0323-495f-8968-2a96fe77d026","Type":"ContainerStarted","Data":"aa412d1a6650ef45444509767fb89ee0b821e0e4d58c72b6ccaf8c2ff7028f37"} Oct 30 00:49:45.006734 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:45.006709 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" event={"ID":"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec","Type":"ContainerStarted","Data":"1a12ec50368b5e785da319a7b92eaade525c57787267e78055f1eaebf67d4bfd"} Oct 30 00:49:45.006848 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:45.006746 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" event={"ID":"20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec","Type":"ContainerStarted","Data":"2f48fa37a39d6a3b26c72226f878a69748dd46c0643518b86300cbc60fb85473"} Oct 30 00:49:45.050348 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:45.050285 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" podStartSLOduration=1.805026054 podStartE2EDuration="3.050271569s" totalImagesPullingTime="1.245245515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:45.049657921 +0000 UTC m=+669.992831112" watchObservedRunningTime="2026-10-30 00:49:45.050271569 +0000 UTC m=+669.993444762" Oct 30 00:49:45.094531 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:45.094476 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" podStartSLOduration=1.891835851 podStartE2EDuration="2.094464045s" totalImagesPullingTime="202.628194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:45.093743117 +0000 UTC m=+670.036916322" watchObservedRunningTime="2026-10-30 00:49:45.094464045 +0000 UTC m=+670.037637235" Oct 30 00:49:45.151962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:45.151940 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:45.627132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:45.627101 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:49.433253 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.433217 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm"] Oct 30 00:49:49.437654 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.437630 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.454179 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.454156 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 00:49:49.473634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.473604 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm"] Oct 30 00:49:49.562968 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.562935 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7r6\" (UniqueName: \"kubernetes.io/projected/34ba07ff-2d49-4483-a644-04ce768e4937-kube-api-access-fx7r6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.563261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.563240 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.563361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.563347 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ba07ff-2d49-4483-a644-04ce768e4937-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.563466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.563454 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.563708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.563686 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.563910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.563893 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665009 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.664967 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665016 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ba07ff-2d49-4483-a644-04ce768e4937-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665061 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665103 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665372 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665277 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665372 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665351 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7r6\" (UniqueName: \"kubernetes.io/projected/34ba07ff-2d49-4483-a644-04ce768e4937-kube-api-access-fx7r6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665482 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665424 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665482 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665466 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.665700 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.665674 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.667822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.667796 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ba07ff-2d49-4483-a644-04ce768e4937-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.667916 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.667844 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ba07ff-2d49-4483-a644-04ce768e4937-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.690638 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.690571 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7r6\" (UniqueName: \"kubernetes.io/projected/34ba07ff-2d49-4483-a644-04ce768e4937-kube-api-access-fx7r6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm\" (UID: \"34ba07ff-2d49-4483-a644-04ce768e4937\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:49.752554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:49.752419 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:50.070046 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.070020 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm"] Oct 30 00:49:50.070978 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:49:50.070949 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ba07ff_2d49_4483_a644_04ce768e4937.slice/crio-102196457b227e9865313f9bfd97600c0e3b0aa40cdcce21206e74893d0e9f04 WatchSource:0}: Error finding container 102196457b227e9865313f9bfd97600c0e3b0aa40cdcce21206e74893d0e9f04: Status 404 returned error can't find the container with id 102196457b227e9865313f9bfd97600c0e3b0aa40cdcce21206e74893d0e9f04 Oct 30 00:49:50.859000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.858876 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9"] Oct 30 00:49:50.863826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.863789 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:50.880239 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.880216 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 00:49:50.895988 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.895964 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9"] Oct 30 00:49:50.976903 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.976879 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e372dca-3376-4d81-9285-f692259df304-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:50.977031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.976927 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ch2\" (UniqueName: \"kubernetes.io/projected/8e372dca-3376-4d81-9285-f692259df304-kube-api-access-w7ch2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:50.977031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.976964 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:50.977244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.977214 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:50.977460 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.977438 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:50.977557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:50.977476 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.040578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.040507 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" event={"ID":"34ba07ff-2d49-4483-a644-04ce768e4937","Type":"ContainerStarted","Data":"7a842bfef7a451c5be016d83dbcba5eaa346b22c3f80e99c5c97f7e2b8c407bf"} Oct 30 00:49:51.040754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.040609 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" event={"ID":"34ba07ff-2d49-4483-a644-04ce768e4937","Type":"ContainerStarted","Data":"102196457b227e9865313f9bfd97600c0e3b0aa40cdcce21206e74893d0e9f04"} Oct 30 00:49:51.079005 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.078971 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e372dca-3376-4d81-9285-f692259df304-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.079225 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.079057 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ch2\" (UniqueName: \"kubernetes.io/projected/8e372dca-3376-4d81-9285-f692259df304-kube-api-access-w7ch2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.079225 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.079123 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.079225 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.079171 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.079225 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.079222 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.079447 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.079252 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.080855 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.080829 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.082021 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.081998 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.082478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.082454 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.083724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.083702 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e372dca-3376-4d81-9285-f692259df304-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.093187 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.093159 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e372dca-3376-4d81-9285-f692259df304-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.093408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.093373 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" podStartSLOduration=1.882439507 podStartE2EDuration="2.09336301s" totalImagesPullingTime="210.923503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:51.092971669 +0000 UTC m=+676.036144857" watchObservedRunningTime="2026-10-30 00:49:51.09336301 +0000 UTC m=+676.036536237" Oct 30 00:49:51.105703 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.105681 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ch2\" (UniqueName: \"kubernetes.io/projected/8e372dca-3376-4d81-9285-f692259df304-kube-api-access-w7ch2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9\" (UID: \"8e372dca-3376-4d81-9285-f692259df304\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.178206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.178102 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:51.487330 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.487300 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9"] Oct 30 00:49:51.487679 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:49:51.487658 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e372dca_3376_4d81_9285_f692259df304.slice/crio-df5da708ca3ce4f0343d7b655500fed305c9f1694e6174689ec9adf341d4d6f3 WatchSource:0}: Error finding container df5da708ca3ce4f0343d7b655500fed305c9f1694e6174689ec9adf341d4d6f3: Status 404 returned error can't find the container with id df5da708ca3ce4f0343d7b655500fed305c9f1694e6174689ec9adf341d4d6f3 Oct 30 00:49:51.753212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:51.753185 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:52.048376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:52.048279 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" event={"ID":"8e372dca-3376-4d81-9285-f692259df304","Type":"ContainerStarted","Data":"c1b96e7336f55b3529857a057dc3c2f0bd6c11f419bbb88ee8c14079ddf14b20"} Oct 30 00:49:52.048376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:52.048341 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" event={"ID":"8e372dca-3376-4d81-9285-f692259df304","Type":"ContainerStarted","Data":"df5da708ca3ce4f0343d7b655500fed305c9f1694e6174689ec9adf341d4d6f3"} Oct 30 00:49:52.104155 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:52.104097 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" podStartSLOduration=1.9099536160000001 podStartE2EDuration="2.104082769s" totalImagesPullingTime="194.129153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:52.103800393 +0000 UTC m=+677.046973583" watchObservedRunningTime="2026-10-30 00:49:52.104082769 +0000 UTC m=+677.047256019" Oct 30 00:49:52.178829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:52.178798 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:49:52.627320 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:52.627283 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:52.641929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:52.641903 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:53.066591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:53.066564 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" Oct 30 00:49:54.152809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:54.152775 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:54.176264 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:54.176237 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:55.080174 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:55.080146 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2rzg2" Oct 30 00:49:59.752958 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:59.752925 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:49:59.765037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:49:59.765009 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:50:00.107323 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:00.107248 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" Oct 30 00:50:01.178535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:01.178490 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:50:01.192165 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:01.192143 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:50:02.117594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:02.117566 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9" Oct 30 00:50:10.222823 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.222789 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747d44f99d-swksm"] Oct 30 00:50:10.223269 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.222988 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-747d44f99d-swksm" podUID="8904dc0e-a270-450b-8234-04ab21b9ed93" containerName="authorino" containerID="cri-o://3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f" gracePeriod=30 Oct 30 00:50:10.469188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.469165 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:50:10.542593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.542570 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8904dc0e-a270-450b-8234-04ab21b9ed93-kube-api-access-wgmgj\" (UniqueName: \"kubernetes.io/projected/8904dc0e-a270-450b-8234-04ab21b9ed93-kube-api-access-wgmgj\") pod \"8904dc0e-a270-450b-8234-04ab21b9ed93\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " Oct 30 00:50:10.542724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.542646 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8904dc0e-a270-450b-8234-04ab21b9ed93-tls-cert\" (UniqueName: \"kubernetes.io/secret/8904dc0e-a270-450b-8234-04ab21b9ed93-tls-cert\") pod \"8904dc0e-a270-450b-8234-04ab21b9ed93\" (UID: \"8904dc0e-a270-450b-8234-04ab21b9ed93\") " Oct 30 00:50:10.544678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.544648 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8904dc0e-a270-450b-8234-04ab21b9ed93-kube-api-access-wgmgj" pod "8904dc0e-a270-450b-8234-04ab21b9ed93" (UID: "8904dc0e-a270-450b-8234-04ab21b9ed93"). InnerVolumeSpecName "kube-api-access-wgmgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:10.553507 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.553483 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8904dc0e-a270-450b-8234-04ab21b9ed93-tls-cert" pod "8904dc0e-a270-450b-8234-04ab21b9ed93" (UID: "8904dc0e-a270-450b-8234-04ab21b9ed93"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:50:10.643670 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.643649 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgmgj\" (UniqueName: \"kubernetes.io/projected/8904dc0e-a270-450b-8234-04ab21b9ed93-kube-api-access-wgmgj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:50:10.643670 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.643671 4905 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8904dc0e-a270-450b-8234-04ab21b9ed93-tls-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:50:10.981387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.981363 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz"] Oct 30 00:50:10.981784 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.981771 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8904dc0e-a270-450b-8234-04ab21b9ed93" containerName="authorino" Oct 30 00:50:10.981830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.981786 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8904dc0e-a270-450b-8234-04ab21b9ed93" containerName="authorino" Oct 30 00:50:10.981872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.981864 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8904dc0e-a270-450b-8234-04ab21b9ed93" containerName="authorino" Oct 30 00:50:10.985369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:10.985355 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.006027 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.006000 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz"] Oct 30 00:50:11.017791 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.017763 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 00:50:11.017989 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.017957 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-h6h5k\"" Oct 30 00:50:11.017989 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.017825 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 00:50:11.147830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.147803 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g825n\" (UniqueName: \"kubernetes.io/projected/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-kube-api-access-g825n\") pod \"ai-gateway-controller-5f85cd9cfc-vz8dz\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.147984 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.147856 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-vz8dz\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.150053 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.150025 4905 generic.go:356] "Generic (PLEG): container finished" podID="8904dc0e-a270-450b-8234-04ab21b9ed93" containerID="3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f" exitCode=0 Oct 30 00:50:11.150157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.150077 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d44f99d-swksm" Oct 30 00:50:11.150157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.150093 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d44f99d-swksm" event={"ID":"8904dc0e-a270-450b-8234-04ab21b9ed93","Type":"ContainerDied","Data":"3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f"} Oct 30 00:50:11.150157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.150136 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d44f99d-swksm" event={"ID":"8904dc0e-a270-450b-8234-04ab21b9ed93","Type":"ContainerDied","Data":"f8dbb99406bde6a362f79148c1e472ba938890f6fd3e0390680cd3632c405657"} Oct 30 00:50:11.150157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.150157 4905 scope.go:122] "RemoveContainer" containerID="3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f" Oct 30 00:50:11.159009 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.158992 4905 scope.go:122] "RemoveContainer" containerID="3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f" Oct 30 00:50:11.159262 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:50:11.159246 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f\": container with ID starting with 3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f not found: ID does not exist" containerID="3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f" Oct 30 00:50:11.159323 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.159272 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f"} err="failed to get container status \"3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f\": rpc error: code = NotFound desc = could not find container \"3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f\": container with ID starting with 3e3609203b05ab1c3147c84df2008e66f0342273fd2e277f0c90507c01a13f9f not found: ID does not exist" Oct 30 00:50:11.204685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.204662 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747d44f99d-swksm"] Oct 30 00:50:11.213234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.213215 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-747d44f99d-swksm"] Oct 30 00:50:11.249341 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.249285 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-vz8dz\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.249786 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.249369 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g825n\" (UniqueName: \"kubernetes.io/projected/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-kube-api-access-g825n\") pod \"ai-gateway-controller-5f85cd9cfc-vz8dz\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.251775 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.251754 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-vz8dz\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.270581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.270563 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g825n\" (UniqueName: \"kubernetes.io/projected/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-kube-api-access-g825n\") pod \"ai-gateway-controller-5f85cd9cfc-vz8dz\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.298088 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.298068 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:11.592814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:11.592785 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz"] Oct 30 00:50:11.593451 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:50:11.593420 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a9d8ce_c0d8_42aa_8a93_f09d160a0745.slice/crio-7e94ca16f8a9b17feef6feac9a76dee5b65a3eeb02ae2defe2c083089cdd0133 WatchSource:0}: Error finding container 7e94ca16f8a9b17feef6feac9a76dee5b65a3eeb02ae2defe2c083089cdd0133: Status 404 returned error can't find the container with id 7e94ca16f8a9b17feef6feac9a76dee5b65a3eeb02ae2defe2c083089cdd0133 Oct 30 00:50:12.158319 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:12.158285 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" event={"ID":"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745","Type":"ContainerStarted","Data":"7e94ca16f8a9b17feef6feac9a76dee5b65a3eeb02ae2defe2c083089cdd0133"} Oct 30 00:50:12.423702 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:50:12.423565 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:50:13.189083 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:13.189047 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8904dc0e-a270-450b-8234-04ab21b9ed93" path="/var/lib/kubelet/pods/8904dc0e-a270-450b-8234-04ab21b9ed93/volumes" Oct 30 00:50:14.172060 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:14.171975 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" event={"ID":"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745","Type":"ContainerStarted","Data":"b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c"} Oct 30 00:50:14.172453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:14.172058 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:14.213495 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:14.213448 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" podStartSLOduration=1.972656671 podStartE2EDuration="4.213436186s" totalImagesPullingTime="2.240779515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:14.212776092 +0000 UTC m=+699.155949287" watchObservedRunningTime="2026-10-30 00:50:14.213436186 +0000 UTC m=+699.156609378" Oct 30 00:50:16.169565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.169533 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds"] Oct 30 00:50:16.173199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.173179 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.208708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.208678 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds"] Oct 30 00:50:16.287887 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.287851 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv"] Oct 30 00:50:16.291578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.291556 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.297747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.297717 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.297865 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.297803 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25db140f-01b8-496e-89c5-c26b749ba26f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.298009 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.297883 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.298009 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.297968 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.298266 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.298242 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25db140f-01b8-496e-89c5-c26b749ba26f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.298368 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.298284 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.298368 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.298310 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.298368 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.298326 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrg6\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-kube-api-access-7wrg6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.298368 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.298348 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.309059 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.309042 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rtbs9\"" Oct 30 00:50:16.316205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.316182 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv"] Oct 30 00:50:16.368746 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.368713 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9tr26"] Oct 30 00:50:16.371612 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.371096 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 00:50:16.373080 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.373054 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.394816 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.394777 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9tr26"] Oct 30 00:50:16.398918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.398882 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25db140f-01b8-496e-89c5-c26b749ba26f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399040 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.398933 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399040 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.398962 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrfgh\" (UniqueName: \"kubernetes.io/projected/3cd3b9f8-446f-4086-be0a-1754d63dc090-kube-api-access-nrfgh\") pod \"payload-pre-processing-7b98bb678c-t8bwv\" (UID: \"3cd3b9f8-446f-4086-be0a-1754d63dc090\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.399040 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.398997 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399065 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrg6\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-kube-api-access-7wrg6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399099 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399148 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3b9f8-446f-4086-be0a-1754d63dc090-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t8bwv\" (UID: \"3cd3b9f8-446f-4086-be0a-1754d63dc090\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.399199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399188 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399260 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25db140f-01b8-496e-89c5-c26b749ba26f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399302 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399344 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.399608 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399485 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.400057 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.399967 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.400057 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.400014 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.400287 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.400092 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.400750 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.400721 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25db140f-01b8-496e-89c5-c26b749ba26f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.401893 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.401873 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25db140f-01b8-496e-89c5-c26b749ba26f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.402495 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.402475 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.423786 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.423734 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.425785 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.425766 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrg6\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-kube-api-access-7wrg6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ts9ds\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.488694 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.488662 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:16.500139 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.500116 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrfgh\" (UniqueName: \"kubernetes.io/projected/3cd3b9f8-446f-4086-be0a-1754d63dc090-kube-api-access-nrfgh\") pod \"payload-pre-processing-7b98bb678c-t8bwv\" (UID: \"3cd3b9f8-446f-4086-be0a-1754d63dc090\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.500213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.500177 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a48816a-4625-4906-b1a2-9876cb2e1365-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9tr26\" (UID: \"3a48816a-4625-4906-b1a2-9876cb2e1365\") " pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.500263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.500248 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3b9f8-446f-4086-be0a-1754d63dc090-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t8bwv\" (UID: \"3cd3b9f8-446f-4086-be0a-1754d63dc090\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.500392 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.500283 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ktw\" (UniqueName: \"kubernetes.io/projected/3a48816a-4625-4906-b1a2-9876cb2e1365-kube-api-access-f6ktw\") pod \"payload-processing-5f78687d5b-9tr26\" (UID: \"3a48816a-4625-4906-b1a2-9876cb2e1365\") " pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.500985 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.500962 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3b9f8-446f-4086-be0a-1754d63dc090-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t8bwv\" (UID: \"3cd3b9f8-446f-4086-be0a-1754d63dc090\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.522022 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.521997 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrfgh\" (UniqueName: \"kubernetes.io/projected/3cd3b9f8-446f-4086-be0a-1754d63dc090-kube-api-access-nrfgh\") pod \"payload-pre-processing-7b98bb678c-t8bwv\" (UID: \"3cd3b9f8-446f-4086-be0a-1754d63dc090\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.601504 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.601471 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a48816a-4625-4906-b1a2-9876cb2e1365-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9tr26\" (UID: \"3a48816a-4625-4906-b1a2-9876cb2e1365\") " pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.601698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.601605 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ktw\" (UniqueName: \"kubernetes.io/projected/3a48816a-4625-4906-b1a2-9876cb2e1365-kube-api-access-f6ktw\") pod \"payload-processing-5f78687d5b-9tr26\" (UID: \"3a48816a-4625-4906-b1a2-9876cb2e1365\") " pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.602152 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.602134 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a48816a-4625-4906-b1a2-9876cb2e1365-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9tr26\" (UID: \"3a48816a-4625-4906-b1a2-9876cb2e1365\") " pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.604298 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.604281 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:16.622555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.622507 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ktw\" (UniqueName: \"kubernetes.io/projected/3a48816a-4625-4906-b1a2-9876cb2e1365-kube-api-access-f6ktw\") pod \"payload-processing-5f78687d5b-9tr26\" (UID: \"3a48816a-4625-4906-b1a2-9876cb2e1365\") " pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.688436 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.688374 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:16.803717 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.803692 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds"] Oct 30 00:50:16.804286 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:50:16.804262 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25db140f_01b8_496e_89c5_c26b749ba26f.slice/crio-cd11744c6d3f80649fdcb7e20778429749eea920fa6bb84cf6b3b6eca3c4be70 WatchSource:0}: Error finding container cd11744c6d3f80649fdcb7e20778429749eea920fa6bb84cf6b3b6eca3c4be70: Status 404 returned error can't find the container with id cd11744c6d3f80649fdcb7e20778429749eea920fa6bb84cf6b3b6eca3c4be70 Oct 30 00:50:16.806639 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.806607 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:50:16.806725 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.806668 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:50:16.806725 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.806700 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:50:16.899435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.899405 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv"] Oct 30 00:50:16.900654 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:50:16.900630 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd3b9f8_446f_4086_be0a_1754d63dc090.slice/crio-fa303cc680917b986966a9caf9ee046b1b43c839194f45c916fffacd7628a03f WatchSource:0}: Error finding container fa303cc680917b986966a9caf9ee046b1b43c839194f45c916fffacd7628a03f: Status 404 returned error can't find the container with id fa303cc680917b986966a9caf9ee046b1b43c839194f45c916fffacd7628a03f Oct 30 00:50:16.983441 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:16.983415 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9tr26"] Oct 30 00:50:16.984830 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:50:16.984807 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a48816a_4625_4906_b1a2_9876cb2e1365.slice/crio-8ebf02f8d8f2b680cf773c627ae582bd0cda955b745518d661cc8b59c2b7fc76 WatchSource:0}: Error finding container 8ebf02f8d8f2b680cf773c627ae582bd0cda955b745518d661cc8b59c2b7fc76: Status 404 returned error can't find the container with id 8ebf02f8d8f2b680cf773c627ae582bd0cda955b745518d661cc8b59c2b7fc76 Oct 30 00:50:17.191485 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:17.191455 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" event={"ID":"3cd3b9f8-446f-4086-be0a-1754d63dc090","Type":"ContainerStarted","Data":"fa303cc680917b986966a9caf9ee046b1b43c839194f45c916fffacd7628a03f"} Oct 30 00:50:17.193104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:17.193073 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" event={"ID":"25db140f-01b8-496e-89c5-c26b749ba26f","Type":"ContainerStarted","Data":"bc77cd748485055be227c44c5a6b37e5b7f064ea27261a6c5776ed082e16b24a"} Oct 30 00:50:17.193256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:17.193127 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" event={"ID":"25db140f-01b8-496e-89c5-c26b749ba26f","Type":"ContainerStarted","Data":"cd11744c6d3f80649fdcb7e20778429749eea920fa6bb84cf6b3b6eca3c4be70"} Oct 30 00:50:17.194319 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:17.194293 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" event={"ID":"3a48816a-4625-4906-b1a2-9876cb2e1365","Type":"ContainerStarted","Data":"8ebf02f8d8f2b680cf773c627ae582bd0cda955b745518d661cc8b59c2b7fc76"} Oct 30 00:50:17.490003 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:17.489928 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:17.490273 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:17.490245 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ts9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:50:17.490385 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:17.490359 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 00:50:18.489373 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:18.489333 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ts9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:50:18.489829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:18.489406 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 00:50:19.490008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:19.489903 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ts9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:50:19.490008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:19.489958 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 00:50:20.213911 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.213816 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" event={"ID":"3cd3b9f8-446f-4086-be0a-1754d63dc090","Type":"ContainerStarted","Data":"d6c7cef881e1b3d41250c9a55663a74bb6a4534af212673d445a789dd75cf9e6"} Oct 30 00:50:20.213911 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.213885 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:20.215330 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.215303 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" event={"ID":"3a48816a-4625-4906-b1a2-9876cb2e1365","Type":"ContainerStarted","Data":"38f3feb826e8504c43c22f667aaec3e326131772e364f77c4dfc2a19c8cd38d8"} Oct 30 00:50:20.215448 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.215350 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:20.268492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.268451 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podStartSLOduration=4.268439252 podStartE2EDuration="4.268439252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:17.247374262 +0000 UTC m=+702.190547454" watchObservedRunningTime="2026-10-30 00:50:20.268439252 +0000 UTC m=+705.211612496" Oct 30 00:50:20.269068 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.269041 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" podStartSLOduration=1.465789728 podStartE2EDuration="4.269031818s" totalImagesPullingTime="2.80324209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:20.267832737 +0000 UTC m=+705.211005929" watchObservedRunningTime="2026-10-30 00:50:20.269031818 +0000 UTC m=+705.212205042" Oct 30 00:50:20.310381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.310341 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" podStartSLOduration=1.5327190910000001 podStartE2EDuration="4.310330191s" totalImagesPullingTime="2.7776111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:20.309415076 +0000 UTC m=+705.252588280" watchObservedRunningTime="2026-10-30 00:50:20.310330191 +0000 UTC m=+705.253503408" Oct 30 00:50:20.489456 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.489425 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ts9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:50:20.489621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:20.489474 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 00:50:21.489929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:21.489889 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ts9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:50:21.490367 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:21.489944 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 00:50:22.489873 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:22.489839 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ts9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:50:22.490047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:22.489891 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 00:50:23.489343 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:23.489313 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ts9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.489500 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:23.489367 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 00:50:24.806604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:24.806568 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:24.807045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:24.807003 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:24.807971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:24.807953 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 00:50:25.188450 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.188380 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:25.456023 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.455933 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8c498c66-26f4b"] Oct 30 00:50:25.460179 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.460159 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:25.479451 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.479427 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8c498c66-26f4b"] Oct 30 00:50:25.516735 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.516711 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzkx\" (UniqueName: \"kubernetes.io/projected/53ff45f8-5864-4727-b72c-b63399c3657c-kube-api-access-fwzkx\") pod \"ai-gateway-controller-5d8c498c66-26f4b\" (UID: \"53ff45f8-5864-4727-b72c-b63399c3657c\") " pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:25.516841 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.516754 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ff45f8-5864-4727-b72c-b63399c3657c-metrics-certs\") pod \"ai-gateway-controller-5d8c498c66-26f4b\" (UID: \"53ff45f8-5864-4727-b72c-b63399c3657c\") " pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:25.617928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.617901 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzkx\" (UniqueName: \"kubernetes.io/projected/53ff45f8-5864-4727-b72c-b63399c3657c-kube-api-access-fwzkx\") pod \"ai-gateway-controller-5d8c498c66-26f4b\" (UID: \"53ff45f8-5864-4727-b72c-b63399c3657c\") " pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:25.618037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.617945 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ff45f8-5864-4727-b72c-b63399c3657c-metrics-certs\") pod \"ai-gateway-controller-5d8c498c66-26f4b\" (UID: \"53ff45f8-5864-4727-b72c-b63399c3657c\") " pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:25.620344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.620317 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ff45f8-5864-4727-b72c-b63399c3657c-metrics-certs\") pod \"ai-gateway-controller-5d8c498c66-26f4b\" (UID: \"53ff45f8-5864-4727-b72c-b63399c3657c\") " pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:25.639270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.639243 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzkx\" (UniqueName: \"kubernetes.io/projected/53ff45f8-5864-4727-b72c-b63399c3657c-kube-api-access-fwzkx\") pod \"ai-gateway-controller-5d8c498c66-26f4b\" (UID: \"53ff45f8-5864-4727-b72c-b63399c3657c\") " pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:25.772659 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:25.772635 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:26.064970 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:26.064942 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8c498c66-26f4b"] Oct 30 00:50:26.065358 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:50:26.064976 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ff45f8_5864_4727_b72c_b63399c3657c.slice/crio-0c6c591ebf866e4403f8d160b1c4666d2e4cdfc04604b0a446684e2c3452b390 WatchSource:0}: Error finding container 0c6c591ebf866e4403f8d160b1c4666d2e4cdfc04604b0a446684e2c3452b390: Status 404 returned error can't find the container with id 0c6c591ebf866e4403f8d160b1c4666d2e4cdfc04604b0a446684e2c3452b390 Oct 30 00:50:26.245037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:26.245004 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" event={"ID":"53ff45f8-5864-4727-b72c-b63399c3657c","Type":"ContainerStarted","Data":"0c6c591ebf866e4403f8d160b1c4666d2e4cdfc04604b0a446684e2c3452b390"} Oct 30 00:50:27.251028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:27.250992 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" event={"ID":"53ff45f8-5864-4727-b72c-b63399c3657c","Type":"ContainerStarted","Data":"ca4d31daec74deff1a681cd08408e84d4b8820b52b8c3378df472058db0e0819"} Oct 30 00:50:27.251401 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:27.251146 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:27.296707 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:27.296660 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" podStartSLOduration=1.928098795 podStartE2EDuration="2.296647826s" totalImagesPullingTime="368.549031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:27.295081292 +0000 UTC m=+712.238254484" watchObservedRunningTime="2026-10-30 00:50:27.296647826 +0000 UTC m=+712.239821038" Oct 30 00:50:31.212001 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.211969 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4"] Oct 30 00:50:31.215716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.215697 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.222918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.222895 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t8bwv" Oct 30 00:50:31.223036 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.222951 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-9tr26" Oct 30 00:50:31.241201 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.241175 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4"] Oct 30 00:50:31.274865 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.274836 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275023 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.274888 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275023 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.275010 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275203 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.275044 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275203 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.275076 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkcd2\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-kube-api-access-pkcd2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275203 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.275100 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.275224 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.275313 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.275458 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.275377 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376085 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376139 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376170 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376212 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376351 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376401 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376427 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376461 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkcd2\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-kube-api-access-pkcd2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376488 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376571 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376725 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.376895 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.376879 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.377100 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.377085 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.377305 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.377279 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.378650 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.378630 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.379199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.379180 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.397610 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.397558 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkcd2\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-kube-api-access-pkcd2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.397770 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.397752 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8zkx4\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.531789 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.531763 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:31.848219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.848195 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4"] Oct 30 00:50:31.848992 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:50:31.848965 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef41398_006e_4f20_aa8b_1a8dd9d4092e.slice/crio-44eb38626d73189b6a2759b3f70e015c05ed5b401efbd4841a26d4d7324845fc WatchSource:0}: Error finding container 44eb38626d73189b6a2759b3f70e015c05ed5b401efbd4841a26d4d7324845fc: Status 404 returned error can't find the container with id 44eb38626d73189b6a2759b3f70e015c05ed5b401efbd4841a26d4d7324845fc Oct 30 00:50:31.851157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.851126 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:50:31.851256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.851202 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:50:31.851256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:31.851243 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:50:32.284807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:32.284769 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" event={"ID":"4ef41398-006e-4f20-aa8b-1a8dd9d4092e","Type":"ContainerStarted","Data":"d8d26e6acb18df183a97d45bd4e2120b75334e27342d6778fac363b8aca5ebbf"} Oct 30 00:50:32.285228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:32.284825 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" event={"ID":"4ef41398-006e-4f20-aa8b-1a8dd9d4092e","Type":"ContainerStarted","Data":"44eb38626d73189b6a2759b3f70e015c05ed5b401efbd4841a26d4d7324845fc"} Oct 30 00:50:32.351769 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:32.351712 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podStartSLOduration=1.351699672 podStartE2EDuration="1.351699672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:32.349998674 +0000 UTC m=+717.293171892" watchObservedRunningTime="2026-10-30 00:50:32.351699672 +0000 UTC m=+717.294872863" Oct 30 00:50:32.532917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:32.532887 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:32.544598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:32.536326 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8zkx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:50:32.544598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:32.536386 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 00:50:33.545544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:33.541931 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8zkx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:50:33.545544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:33.541990 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 00:50:34.537091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:34.537057 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8zkx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:50:34.537405 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:34.537362 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 00:50:35.546033 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:35.545948 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8zkx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:50:35.546616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:35.546585 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 00:50:36.549921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:36.549879 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8zkx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:50:36.550300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:36.549956 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 00:50:37.533014 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:37.532975 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8zkx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:50:37.533191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:37.533054 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 00:50:38.258098 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.258069 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8c498c66-26f4b" Oct 30 00:50:38.336853 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.336812 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz"] Oct 30 00:50:38.337092 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.337064 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" podUID="c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" containerName="manager" containerID="cri-o://b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c" gracePeriod=30 Oct 30 00:50:38.533058 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.532990 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8zkx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:50:38.533181 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.533053 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 00:50:38.588034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.588013 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:38.665200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.665168 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-metrics-certs\") pod \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " Oct 30 00:50:38.665357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.665268 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-kube-api-access-g825n\" (UniqueName: \"kubernetes.io/projected/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-kube-api-access-g825n\") pod \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\" (UID: \"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745\") " Oct 30 00:50:38.667350 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.667319 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-metrics-certs" pod "c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" (UID: "c0a9d8ce-c0d8-42aa-8a93-f09d160a0745"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:50:38.667480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.667454 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-kube-api-access-g825n" pod "c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" (UID: "c0a9d8ce-c0d8-42aa-8a93-f09d160a0745"). InnerVolumeSpecName "kube-api-access-g825n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:38.766975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.766942 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:50:38.766975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:38.766972 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g825n\" (UniqueName: \"kubernetes.io/projected/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745-kube-api-access-g825n\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:50:39.324147 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.324113 4905 generic.go:356] "Generic (PLEG): container finished" podID="c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" containerID="b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c" exitCode=0 Oct 30 00:50:39.324625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.324179 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" event={"ID":"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745","Type":"ContainerDied","Data":"b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c"} Oct 30 00:50:39.324625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.324191 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" Oct 30 00:50:39.324625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.324222 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz" event={"ID":"c0a9d8ce-c0d8-42aa-8a93-f09d160a0745","Type":"ContainerDied","Data":"7e94ca16f8a9b17feef6feac9a76dee5b65a3eeb02ae2defe2c083089cdd0133"} Oct 30 00:50:39.324625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.324246 4905 scope.go:122] "RemoveContainer" containerID="b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c" Oct 30 00:50:39.333963 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.333944 4905 scope.go:122] "RemoveContainer" containerID="b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c" Oct 30 00:50:39.334261 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:50:39.334234 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c\": container with ID starting with b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c not found: ID does not exist" containerID="b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c" Oct 30 00:50:39.334324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.334269 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c"} err="failed to get container status \"b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c\": rpc error: code = NotFound desc = could not find container \"b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c\": container with ID starting with b0e3d00054f7681cacda55e947519ea0864a46e9e29ffb06d4abddcc327cd32c not found: ID does not exist" Oct 30 00:50:39.379864 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.379837 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz"] Oct 30 00:50:39.393312 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.393290 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz"] Oct 30 00:50:39.834624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.834591 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:39.835153 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.835133 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:39.835868 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:39.835849 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 00:50:41.189630 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:50:41.189593 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" path="/var/lib/kubelet/pods/c0a9d8ce-c0d8-42aa-8a93-f09d160a0745/volumes" Oct 30 00:50:42.412986 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:50:42.412945 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:51:12.412377 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:51:12.412341 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:51:29.639245 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.639212 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f"] Oct 30 00:51:29.639845 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.639826 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" containerName="manager" Oct 30 00:51:29.639924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.639848 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" containerName="manager" Oct 30 00:51:29.640000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.639988 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a9d8ce-c0d8-42aa-8a93-f09d160a0745" containerName="manager" Oct 30 00:51:29.643012 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.642993 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.660037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.660017 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7d2b75-openshift-default-dockercfg-pft72\"" Oct 30 00:51:29.675063 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.675038 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f"] Oct 30 00:51:29.774800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.774767 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-data\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.774951 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.774823 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-token\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.775024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.774963 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-envoy\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.775024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.775007 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psl4t\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-kube-api-access-psl4t\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.775127 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.775028 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-credential-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.775127 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.775067 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-certs\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.775234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.775188 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.775234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.775203 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh"] Oct 30 00:51:29.775334 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.775240 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-podinfo\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.775334 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.775264 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istiod-ca-cert\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.779131 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.779113 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.798682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.798662 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8cf33a-openshift-default-dockercfg-j7d2l\"" Oct 30 00:51:29.805084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.805062 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh"] Oct 30 00:51:29.875815 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.875775 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-token\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.875966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.875829 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.875966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.875873 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-envoy\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.875966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.875911 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psl4t\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-kube-api-access-psl4t\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876144 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.875988 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-credential-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876144 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876054 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-certs\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876144 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876099 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876144 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876134 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876169 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876313 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876471 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876349 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876471 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876377 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-credential-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876471 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876446 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-podinfo\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876471 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876447 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-certs\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876474 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istiod-ca-cert\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876505 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876609 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-data\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876642 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876667 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snk9\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-kube-api-access-8snk9\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876695 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.876938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876868 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.876938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.876881 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-data\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.877032 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.877014 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istiod-ca-cert\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.878367 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.878342 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-envoy\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.878912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.878896 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-podinfo\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.897138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.897079 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psl4t\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-kube-api-access-psl4t\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.897301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.897284 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-token\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.914950 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.914926 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s"] Oct 30 00:51:29.930863 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.930845 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:29.958763 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.958741 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:29.978250 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978225 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978340 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978447 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978382 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978447 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978418 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978472 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978542 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978680 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978583 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978680 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978609 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snk9\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-kube-api-access-8snk9\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.978680 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978641 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.979837 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978929 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.979837 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.978948 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.979837 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.979193 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.979837 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.979211 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.979837 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.979798 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.981467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.981446 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.982832 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.982812 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:29.985046 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.985026 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s"] Oct 30 00:51:29.999462 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:29.999437 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3c2596-openshift-default-dockercfg-mts5l\"" Oct 30 00:51:30.009990 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.009125 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snk9\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-kube-api-access-8snk9\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:30.012061 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.012036 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:30.079736 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079708 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.079736 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079738 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.080031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079756 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33408e5-7fea-4575-aabb-09bee5bbb41e-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.080031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079810 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.080031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079825 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.080031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079845 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.080031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079880 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.080031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079896 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.080031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.079936 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgdk\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-kube-api-access-jwgdk\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.096182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.096157 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:30.141584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.139739 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz"] Oct 30 00:51:30.144615 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.144589 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.177269 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz"] Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.180745 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgdk\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-kube-api-access-jwgdk\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.180896 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.180944 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.180971 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33408e5-7fea-4575-aabb-09bee5bbb41e-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.181110 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.181136 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.181182 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.181258 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.181282 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.181412 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.181721 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.182095 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.182589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.182140 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.183383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.182685 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33408e5-7fea-4575-aabb-09bee5bbb41e-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.185983 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.185957 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.187493 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.187466 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.213267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.208335 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.213267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.210867 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b565ec-openshift-default-dockercfg-sl86j\"" Oct 30 00:51:30.213267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.211995 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgdk\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-kube-api-access-jwgdk\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.247980 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.247951 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.281930 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282057 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twsfd\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-kube-api-access-twsfd\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282143 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282171 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282323 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282444 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282492 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.282557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282535 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.283051 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.282719 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.302738 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.302623 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f"] Oct 30 00:51:30.310752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.306954 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8"] Oct 30 00:51:30.311020 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:51:30.310981 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9eed20_b131_46a4_84e2_e1dfa732bfdb.slice/crio-5633d7d556f15d30803faf8b77084971cec2f6d8865a44e3587e61bb997d5976 WatchSource:0}: Error finding container 5633d7d556f15d30803faf8b77084971cec2f6d8865a44e3587e61bb997d5976: Status 404 returned error can't find the container with id 5633d7d556f15d30803faf8b77084971cec2f6d8865a44e3587e61bb997d5976 Oct 30 00:51:30.316568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.316354 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.318828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.318798 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.318914 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.318871 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.318975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.318913 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.337340 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.337074 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-da76a2-openshift-default-dockercfg-8l8nk\"" Oct 30 00:51:30.345232 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.345209 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8"] Oct 30 00:51:30.383783 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.383722 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.383783 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.383766 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.383952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.383803 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.383952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.383876 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.383952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.383905 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.383952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.383935 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.384090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.383968 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.384090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384004 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twsfd\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-kube-api-access-twsfd\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.384090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384033 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.384090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384058 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.384090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384084 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.384262 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384138 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.384262 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384171 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.384262 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384198 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.384262 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384246 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.384386 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384298 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.384386 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384355 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.384448 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384382 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqpg\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-kube-api-access-jnqpg\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.384795 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.384772 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.385196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.385175 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.385587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.385566 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.385708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.385566 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.385869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.385847 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.387120 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.387080 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.387575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.387559 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.408735 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.408712 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh"] Oct 30 00:51:30.410866 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.410825 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.413409 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.413365 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.413555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.413460 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.413555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.413505 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.415255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.415236 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twsfd\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-kube-api-access-twsfd\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.431312 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.431283 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x"] Oct 30 00:51:30.437972 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.437955 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.465472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.465266 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:30.465792 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.465773 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5a344d-openshift-default-dockercfg-7z8cg\"" Oct 30 00:51:30.471192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.471138 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x"] Oct 30 00:51:30.485457 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.485428 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.485666 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.485648 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.485787 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.485774 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.485904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.485890 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486012 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486000 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486111 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486098 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.486228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486214 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486308 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486322 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486425 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqpg\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-kube-api-access-jnqpg\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486458 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrsm\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-kube-api-access-msrsm\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486489 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486539 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486584 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486611 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486652 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486681 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486714 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.486821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.486739 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.487650 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.487627 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.488049 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.488029 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.490321 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.490302 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.490877 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.490859 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.492641 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.492490 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.498922 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.498865 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.516076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.516054 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqpg\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-kube-api-access-jnqpg\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.518480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.518443 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.589990 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590070 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590130 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590181 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrsm\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-kube-api-access-msrsm\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590233 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590305 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590345 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590418 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.590448 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.591550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.591352 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.593600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.592435 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.593600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.592874 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.608543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.608352 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s"] Oct 30 00:51:30.628374 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.615502 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.628374 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.616239 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.640303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.633594 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5"] Oct 30 00:51:30.640303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.639244 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.640303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.639448 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.640303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.639626 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.649625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.648857 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5"] Oct 30 00:51:30.649625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.648980 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.650286 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.650108 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:30.657288 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.657235 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" event={"ID":"ed9eed20-b131-46a4-84e2-e1dfa732bfdb","Type":"ContainerStarted","Data":"637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180"} Oct 30 00:51:30.657454 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.657437 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" event={"ID":"ed9eed20-b131-46a4-84e2-e1dfa732bfdb","Type":"ContainerStarted","Data":"5633d7d556f15d30803faf8b77084971cec2f6d8865a44e3587e61bb997d5976"} Oct 30 00:51:30.666294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.666273 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.673940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.668020 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-298ddc-openshift-default-dockercfg-lcv79\"" Oct 30 00:51:30.673940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.668172 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.673940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.670419 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.673940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.672850 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" event={"ID":"151a86fd-2313-46ba-9470-7fc0c8f22d1b","Type":"ContainerStarted","Data":"909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d"} Oct 30 00:51:30.673940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.672908 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" event={"ID":"151a86fd-2313-46ba-9470-7fc0c8f22d1b","Type":"ContainerStarted","Data":"695bd8c50c97d07d7d8e920f553b5cd05745d9695c6a671c94e3affc4916b383"} Oct 30 00:51:30.675929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.675912 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrsm\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-kube-api-access-msrsm\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.789184 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.788957 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:30.834485 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.834432 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podStartSLOduration=1.834414284 podStartE2EDuration="1.834414284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:30.830465316 +0000 UTC m=+775.773638509" watchObservedRunningTime="2026-10-30 00:51:30.834414284 +0000 UTC m=+775.777587473" Oct 30 00:51:30.836803 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.836779 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz"] Oct 30 00:51:30.844813 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:51:30.844785 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d11eb5_43ca_41a9_b945_3219c8e0bcc7.slice/crio-300bcbf233dc808445ba96a2fe247fa213db12fd8179e1df4a1e6fbe53b14786 WatchSource:0}: Error finding container 300bcbf233dc808445ba96a2fe247fa213db12fd8179e1df4a1e6fbe53b14786: Status 404 returned error can't find the container with id 300bcbf233dc808445ba96a2fe247fa213db12fd8179e1df4a1e6fbe53b14786 Oct 30 00:51:30.861764 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.860795 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.861764 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.860889 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.861764 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.860936 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:30.884697 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.884586 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podStartSLOduration=1.884567867 podStartE2EDuration="1.884567867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:30.883639736 +0000 UTC m=+775.826812927" watchObservedRunningTime="2026-10-30 00:51:30.884567867 +0000 UTC m=+775.827741061" Oct 30 00:51:30.889218 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.889127 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-envoy\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.889378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.889227 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-data\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.889378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.889349 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-certs\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.889504 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.889411 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.891137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.889585 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4f7\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-kube-api-access-dw4f7\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.891137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.890250 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6392db76-3331-49b5-9bf3-055e24d0f203-istio-podinfo\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.891137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.890313 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-istio-token\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.891137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.890425 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-credential-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.891137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.890468 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6392db76-3331-49b5-9bf3-055e24d0f203-istiod-ca-cert\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:30.976544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.971335 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.998738 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-data\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.998842 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-certs\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.998878 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.998979 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4f7\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-kube-api-access-dw4f7\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.999039 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6392db76-3331-49b5-9bf3-055e24d0f203-istio-podinfo\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.999066 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-istio-token\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.999140 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-credential-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.999169 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6392db76-3331-49b5-9bf3-055e24d0f203-istiod-ca-cert\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:30.999210 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-envoy\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.000305 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-data\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.001441 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-certs\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.001691 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.005007 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-credential-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.019196 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6392db76-3331-49b5-9bf3-055e24d0f203-istiod-ca-cert\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.020611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.020163 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6392db76-3331-49b5-9bf3-055e24d0f203-istio-podinfo\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.030246 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.030191 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4f7\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-kube-api-access-dw4f7\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.050631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.049240 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-envoy\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.054961 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.054352 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-istio-token\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.096631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.096555 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:31.102983 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.102957 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:31.118756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.115625 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8"] Oct 30 00:51:31.122160 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.122120 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.122259 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.122208 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.122329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.122255 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.209032 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.208113 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x"] Oct 30 00:51:31.216181 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.215775 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.216181 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.215837 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.216181 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.215874 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.321112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.321086 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:31.664623 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.664591 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5"] Oct 30 00:51:31.664983 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:51:31.664896 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6392db76_3331_49b5_9bf3_055e24d0f203.slice/crio-1d29649c4bb1087132d0325b77613026ae79c3c3047dba19f93e19b2e368e345 WatchSource:0}: Error finding container 1d29649c4bb1087132d0325b77613026ae79c3c3047dba19f93e19b2e368e345: Status 404 returned error can't find the container with id 1d29649c4bb1087132d0325b77613026ae79c3c3047dba19f93e19b2e368e345 Oct 30 00:51:31.667902 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.667864 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.668002 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.667955 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.668046 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.668029 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:51:31.683371 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.683342 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" event={"ID":"c33408e5-7fea-4575-aabb-09bee5bbb41e","Type":"ContainerStarted","Data":"61888bfb77b2d6031d167ef0ff5916473beb626ce0f1a8dc5221e44efaf3b1e9"} Oct 30 00:51:31.683477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.683407 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" event={"ID":"c33408e5-7fea-4575-aabb-09bee5bbb41e","Type":"ContainerStarted","Data":"b327aed0b4b1230669513c92eed8979e5f61657dca7eb8f9c211393f158aabe0"} Oct 30 00:51:31.685567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.685513 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" event={"ID":"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7","Type":"ContainerStarted","Data":"ff56d0289a87d165bbfcd464dd8c32411586c51736d0c07fe3395192f5787d85"} Oct 30 00:51:31.685660 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.685592 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" event={"ID":"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7","Type":"ContainerStarted","Data":"300bcbf233dc808445ba96a2fe247fa213db12fd8179e1df4a1e6fbe53b14786"} Oct 30 00:51:31.687146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.687121 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" event={"ID":"feaf92d1-706a-4379-b59b-adf69e1fdaf5","Type":"ContainerStarted","Data":"f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312"} Oct 30 00:51:31.687262 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.687171 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" event={"ID":"feaf92d1-706a-4379-b59b-adf69e1fdaf5","Type":"ContainerStarted","Data":"ccf01b961bfad1f63a419fad16912de5b144e1d30e438dfdc6a6baab4094c34b"} Oct 30 00:51:31.688940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.688913 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" event={"ID":"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1","Type":"ContainerStarted","Data":"9db5876a683516c0129d0c3344125381824bca37d5cb868c9bebf078b52107be"} Oct 30 00:51:31.689019 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.688956 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" event={"ID":"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1","Type":"ContainerStarted","Data":"ab622b5808beb878ab4f90af4038f0bd5ea061e97896c2086e15af8cc2a89998"} Oct 30 00:51:31.690510 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.690486 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" event={"ID":"6392db76-3331-49b5-9bf3-055e24d0f203","Type":"ContainerStarted","Data":"1d29649c4bb1087132d0325b77613026ae79c3c3047dba19f93e19b2e368e345"} Oct 30 00:51:31.690886 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.690866 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:31.691921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.691905 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:51:31.730151 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.730099 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podStartSLOduration=2.730085826 podStartE2EDuration="2.730085826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:31.728998016 +0000 UTC m=+776.672171207" watchObservedRunningTime="2026-10-30 00:51:31.730085826 +0000 UTC m=+776.673259017" Oct 30 00:51:31.789708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.789660 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:31.887342 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.887286 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podStartSLOduration=1.887268272 podStartE2EDuration="1.887268272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:31.839531833 +0000 UTC m=+776.782705020" watchObservedRunningTime="2026-10-30 00:51:31.887268272 +0000 UTC m=+776.830441466" Oct 30 00:51:31.887538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.887439 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podStartSLOduration=1.887430242 podStartE2EDuration="1.887430242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:31.885181977 +0000 UTC m=+776.828355168" watchObservedRunningTime="2026-10-30 00:51:31.887430242 +0000 UTC m=+776.830603436" Oct 30 00:51:31.934371 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.934265 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podStartSLOduration=1.934251674 podStartE2EDuration="1.934251674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:31.933074893 +0000 UTC m=+776.876248084" watchObservedRunningTime="2026-10-30 00:51:31.934251674 +0000 UTC m=+776.877424950" Oct 30 00:51:31.964653 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:31.964626 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:32.248622 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.248588 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:32.254740 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.254717 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:32.465757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.465725 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:32.471890 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.471864 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:32.650591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.650487 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:32.656499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.656474 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:32.699491 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.699456 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" event={"ID":"6392db76-3331-49b5-9bf3-055e24d0f203","Type":"ContainerStarted","Data":"04909fc8e66b995e3b0d85b10ff8e024e884703509bf2b86275a93b164d631df"} Oct 30 00:51:32.700006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.699776 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:32.700822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.700431 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:32.700822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.700556 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:32.701636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.701417 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:32.701636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.701558 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:51:32.701636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.701606 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:51:32.702168 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.702148 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:51:32.702255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.702208 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:51:32.796729 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.796698 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:32.936362 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:32.936262 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podStartSLOduration=2.93624809 podStartE2EDuration="2.93624809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:32.936031415 +0000 UTC m=+777.879204617" watchObservedRunningTime="2026-10-30 00:51:32.93624809 +0000 UTC m=+777.879421281" Oct 30 00:51:33.322424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:33.322393 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:33.328227 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:33.328203 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:33.711447 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:33.711119 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:33.711892 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:33.711716 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:33.711956 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:33.711874 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:51:33.712540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:33.712500 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:51:42.442898 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:51:42.442858 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:51:43.256187 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.256153 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654"] Oct 30 00:51:43.261271 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.261250 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.291416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.291384 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5a344d-dockercfg-pznhc\"" Oct 30 00:51:43.291416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.291410 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5a344d\"" Oct 30 00:51:43.294773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.294750 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654"] Oct 30 00:51:43.392950 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.391721 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r526b\" (UniqueName: \"kubernetes.io/projected/10ae9f35-3087-4a5a-ab61-7c856cb32928-kube-api-access-r526b\") pod \"payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.392950 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.391777 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ae9f35-3087-4a5a-ab61-7c856cb32928-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.437578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.436821 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6"] Oct 30 00:51:43.442767 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.442743 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6"] Oct 30 00:51:43.442891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.442855 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.493171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.493145 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9569e49a-21e9-447b-b435-ff3cb63a323c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.493582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.493259 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2qt\" (UniqueName: \"kubernetes.io/projected/9569e49a-21e9-447b-b435-ff3cb63a323c-kube-api-access-rf2qt\") pod \"payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.493582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.493298 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r526b\" (UniqueName: \"kubernetes.io/projected/10ae9f35-3087-4a5a-ab61-7c856cb32928-kube-api-access-r526b\") pod \"payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.493582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.493326 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ae9f35-3087-4a5a-ab61-7c856cb32928-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.494007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.493991 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ae9f35-3087-4a5a-ab61-7c856cb32928-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.536635 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.536578 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r526b\" (UniqueName: \"kubernetes.io/projected/10ae9f35-3087-4a5a-ab61-7c856cb32928-kube-api-access-r526b\") pod \"payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.577414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.577393 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:43.594467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.594436 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9569e49a-21e9-447b-b435-ff3cb63a323c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.594621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.594578 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2qt\" (UniqueName: \"kubernetes.io/projected/9569e49a-21e9-447b-b435-ff3cb63a323c-kube-api-access-rf2qt\") pod \"payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.595272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.595250 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9569e49a-21e9-447b-b435-ff3cb63a323c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.625871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.625844 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2qt\" (UniqueName: \"kubernetes.io/projected/9569e49a-21e9-447b-b435-ff3cb63a323c-kube-api-access-rf2qt\") pod \"payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.758474 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.758434 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:43.877287 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:51:43.877260 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ae9f35_3087_4a5a_ab61_7c856cb32928.slice/crio-040513bca08d85c8416e9af7f3c684d0fc46e8263aded44e49ac178a83a6ace6 WatchSource:0}: Error finding container 040513bca08d85c8416e9af7f3c684d0fc46e8263aded44e49ac178a83a6ace6: Status 404 returned error can't find the container with id 040513bca08d85c8416e9af7f3c684d0fc46e8263aded44e49ac178a83a6ace6 Oct 30 00:51:43.879582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:43.879561 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654"] Oct 30 00:51:44.098315 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.098288 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6"] Oct 30 00:51:44.098757 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:51:44.098733 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9569e49a_21e9_447b_b435_ff3cb63a323c.slice/crio-3c67f060e2e62c527d3c1552d2695e0eff6cd5abf97f548f2a15f39c859be95c WatchSource:0}: Error finding container 3c67f060e2e62c527d3c1552d2695e0eff6cd5abf97f548f2a15f39c859be95c: Status 404 returned error can't find the container with id 3c67f060e2e62c527d3c1552d2695e0eff6cd5abf97f548f2a15f39c859be95c Oct 30 00:51:44.840130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.840092 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" event={"ID":"9569e49a-21e9-447b-b435-ff3cb63a323c","Type":"ContainerStarted","Data":"f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc"} Oct 30 00:51:44.840630 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.840161 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:51:44.840630 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.840182 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" event={"ID":"9569e49a-21e9-447b-b435-ff3cb63a323c","Type":"ContainerStarted","Data":"3c67f060e2e62c527d3c1552d2695e0eff6cd5abf97f548f2a15f39c859be95c"} Oct 30 00:51:44.841902 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.841871 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" event={"ID":"10ae9f35-3087-4a5a-ab61-7c856cb32928","Type":"ContainerStarted","Data":"7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098"} Oct 30 00:51:44.842042 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.841922 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" event={"ID":"10ae9f35-3087-4a5a-ab61-7c856cb32928","Type":"ContainerStarted","Data":"040513bca08d85c8416e9af7f3c684d0fc46e8263aded44e49ac178a83a6ace6"} Oct 30 00:51:44.842042 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.842005 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:44.924163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.924119 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" podStartSLOduration=1.9241079079999999 podStartE2EDuration="1.924107908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:44.923143305 +0000 UTC m=+789.866316499" watchObservedRunningTime="2026-10-30 00:51:44.924107908 +0000 UTC m=+789.867281132" Oct 30 00:51:44.924407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:44.924387 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" podStartSLOduration=1.924383834 podStartE2EDuration="1.924383834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:44.882714774 +0000 UTC m=+789.825887966" watchObservedRunningTime="2026-10-30 00:51:44.924383834 +0000 UTC m=+789.867557025" Oct 30 00:51:55.886604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:55.884906 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 00:51:55.886604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:51:55.885989 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 00:52:02.733593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:02.732689 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:02.733593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:02.732753 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:03.741974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.741661 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:03.741974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.741716 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:03.741974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.741746 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:03.741974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.741746 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:04.284132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.742641 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:04.284132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.742681 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:04.284132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.743628 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:04.284132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:03.743679 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:04.748690 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:04.748649 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:04.761199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:04.748726 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.434295 4905 generic.go:356] "Generic (PLEG): container finished" podID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerID="61888bfb77b2d6031d167ef0ff5916473beb626ce0f1a8dc5221e44efaf3b1e9" exitCode=0 Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.434378 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" event={"ID":"c33408e5-7fea-4575-aabb-09bee5bbb41e","Type":"ContainerDied","Data":"61888bfb77b2d6031d167ef0ff5916473beb626ce0f1a8dc5221e44efaf3b1e9"} Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.434788 4905 scope.go:122] "RemoveContainer" containerID="61888bfb77b2d6031d167ef0ff5916473beb626ce0f1a8dc5221e44efaf3b1e9" Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.436268 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.436299 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" event={"ID":"feaf92d1-706a-4379-b59b-adf69e1fdaf5","Type":"ContainerDied","Data":"f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312"} Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.436353 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.436280 4905 generic.go:356] "Generic (PLEG): container finished" podID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerID="f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312" exitCode=0 Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.436402 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.436684 4905 scope.go:122] "RemoveContainer" containerID="f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312" Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.437887 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.437956 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:06.454901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:06.437991 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:07.461651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.453497 4905 generic.go:356] "Generic (PLEG): container finished" podID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerID="637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180" exitCode=0 Oct 30 00:52:07.461651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.453601 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" event={"ID":"ed9eed20-b131-46a4-84e2-e1dfa732bfdb","Type":"ContainerDied","Data":"637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180"} Oct 30 00:52:07.461651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.454063 4905 scope.go:122] "RemoveContainer" containerID="637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180" Oct 30 00:52:07.461651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.456012 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:07.461651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.456095 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:07.461651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.456138 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:07.461651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.460276 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" event={"ID":"c33408e5-7fea-4575-aabb-09bee5bbb41e","Type":"ContainerStarted","Data":"9e6cb3b614c2fc7efdcd39fed7900a943ca82188078f66cc84d613603ed2c3e5"} Oct 30 00:52:07.501545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.466749 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" event={"ID":"feaf92d1-706a-4379-b59b-adf69e1fdaf5","Type":"ContainerStarted","Data":"9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149"} Oct 30 00:52:07.652765 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.652136 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:52:07.652765 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.652293 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:07.652765 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:07.652352 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:08.250543 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:52:08.241931 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d11eb5_43ca_41a9_b945_3219c8e0bcc7.slice/crio-conmon-ff56d0289a87d165bbfcd464dd8c32411586c51736d0c07fe3395192f5787d85.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@4-14133-0.service\": RecentStats: unable to find data in memory cache]" Oct 30 00:52:08.284572 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.283456 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:52:08.284572 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.283590 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:08.284572 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.283630 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:08.518114 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.517298 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" event={"ID":"ed9eed20-b131-46a4-84e2-e1dfa732bfdb","Type":"ContainerStarted","Data":"9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae"} Oct 30 00:52:08.520920 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.520195 4905 generic.go:356] "Generic (PLEG): container finished" podID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerID="ff56d0289a87d165bbfcd464dd8c32411586c51736d0c07fe3395192f5787d85" exitCode=0 Oct 30 00:52:08.520920 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.520569 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" event={"ID":"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7","Type":"ContainerDied","Data":"ff56d0289a87d165bbfcd464dd8c32411586c51736d0c07fe3395192f5787d85"} Oct 30 00:52:08.521115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.521047 4905 scope.go:122] "RemoveContainer" containerID="ff56d0289a87d165bbfcd464dd8c32411586c51736d0c07fe3395192f5787d85" Oct 30 00:52:08.523292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.522647 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:08.523292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.522705 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:08.523292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.522734 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:08.667567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.662445 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:08.667567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.662497 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:08.994569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:08.968780 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:52:09.287583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:09.265113 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:09.287583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:09.265269 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:09.659717 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:09.659685 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:09.660247 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:09.660215 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:10.089605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:09.985661 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:10.089605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:09.985741 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:10.250667 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.250631 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:10.250972 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.250928 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:10.668344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.668308 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:10.669394 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.669340 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:10.688177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.688135 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" event={"ID":"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7","Type":"ContainerStarted","Data":"fa08e786d89b46ab1ef9ec92b87cf7a1aca1fff96678132bbd1ba4dece0a4210"} Oct 30 00:52:10.690939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.690912 4905 generic.go:356] "Generic (PLEG): container finished" podID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerID="909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d" exitCode=0 Oct 30 00:52:10.691113 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.691096 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" event={"ID":"151a86fd-2313-46ba-9470-7fc0c8f22d1b","Type":"ContainerDied","Data":"909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d"} Oct 30 00:52:10.691569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.691552 4905 scope.go:122] "RemoveContainer" containerID="909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d" Oct 30 00:52:10.693573 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.693540 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:10.693743 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.693721 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:10.693894 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.693875 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:10.969567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.969455 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:10.969814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:10.969787 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:11.266657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.248416 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:11.266657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.248480 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:11.469539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.468557 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:52:11.469539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.468993 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:11.469539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.469050 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:11.663596 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.662917 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:11.663596 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.662980 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:11.743543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.728228 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" event={"ID":"151a86fd-2313-46ba-9470-7fc0c8f22d1b","Type":"ContainerStarted","Data":"9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc"} Oct 30 00:52:11.962041 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.961948 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:11.962317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:11.962286 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:12.100742 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.100712 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:52:12.101074 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.101052 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.101228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.101206 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:12.255951 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.255915 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.256229 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.256199 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:12.466124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.466091 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.466423 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.466381 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:12.653602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.650972 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.653602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.651066 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:12.700955 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:52:12.700884 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:52:12.968489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.967959 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.968489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:12.968020 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:13.111542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.110670 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.111542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.110815 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:13.249957 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.249923 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.250223 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.250195 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:13.477577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.472927 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.477577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.472982 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:13.660600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.656922 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.660600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.656982 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:13.965510 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.964986 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.965510 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:13.965052 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:14.106115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.105265 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.106115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.105321 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:14.249779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.249744 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.250071 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.250037 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:14.477462 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.477427 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.477757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.477729 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:14.660479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.660045 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.660479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.660105 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 00:52:14.964977 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.964893 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.965225 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:14.965196 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:15.101975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.101110 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:15.101975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.101169 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:15.249383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.248943 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:52:15.249383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.249009 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 00:52:15.488239 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.485217 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:15.488239 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.485274 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:15.639648 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.634532 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns"] Oct 30 00:52:15.670542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.665904 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns"] Oct 30 00:52:15.670542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.666024 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.909856 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.909943 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.909988 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202a9b97-fc86-4e8d-b3e8-380942de9f32-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.910078 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.910105 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.910148 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpf6r\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-kube-api-access-wpf6r\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.910184 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.910261 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.936616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.910296 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:15.965185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.959470 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:15.965185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.959548 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:15.994546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.967470 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:52:15.995084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:15.995061 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.022796 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023194 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023242 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023285 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023348 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023390 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202a9b97-fc86-4e8d-b3e8-380942de9f32-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023478 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023502 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023568 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpf6r\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-kube-api-access-wpf6r\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.023605 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.024107 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024894 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.024344 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.024954 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.024893 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.055383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.032769 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.055383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.033240 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.055383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.033389 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202a9b97-fc86-4e8d-b3e8-380942de9f32-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.063883 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.062417 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.100246 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.099716 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:16.100246 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.099772 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:16.101005 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.100964 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.226600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.195602 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpf6r\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-kube-api-access-wpf6r\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.348544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.347769 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:52:16.367569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.349115 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:52:16.394103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.384431 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:16.394103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.385883 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 00:52:16.485592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.466679 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:16.485592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.466735 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:16.757492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.755621 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:52:16.998590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.961003 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:52:16.998590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:16.961065 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:52:17.224232 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.224157 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:17.224452 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.224427 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:17.451564 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.442279 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns"] Oct 30 00:52:17.452203 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:52:17.452096 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202a9b97_fc86_4e8d_b3e8_380942de9f32.slice/crio-4fbc5e7c6f917ada31f62422f2d5c87b492d2a6bfa5ea01bbc7b7cb18dca19ce WatchSource:0}: Error finding container 4fbc5e7c6f917ada31f62422f2d5c87b492d2a6bfa5ea01bbc7b7cb18dca19ce: Status 404 returned error can't find the container with id 4fbc5e7c6f917ada31f62422f2d5c87b492d2a6bfa5ea01bbc7b7cb18dca19ce Oct 30 00:52:17.482200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.466370 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:17.482200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.466416 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:17.482200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.477389 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:17.482200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.477477 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:17.487886 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.485646 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:17.702834 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.700606 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:52:17.702834 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:17.701745 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:52:18.105593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.102756 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:18.105593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.102817 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:18.393797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.393713 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:52:18.416489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.416453 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" event={"ID":"202a9b97-fc86-4e8d-b3e8-380942de9f32","Type":"ContainerStarted","Data":"27782b6555faa15ee8fc3fc11661f4a46f5e309bf03c420ccfcb7c07b93ee8fc"} Oct 30 00:52:18.416759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.416741 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" event={"ID":"202a9b97-fc86-4e8d-b3e8-380942de9f32","Type":"ContainerStarted","Data":"4fbc5e7c6f917ada31f62422f2d5c87b492d2a6bfa5ea01bbc7b7cb18dca19ce"} Oct 30 00:52:18.416893 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.416882 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 00:52:18.554491 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.554450 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:52:18.623576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.600607 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:52:18.623576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:18.609260 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podStartSLOduration=3.609228806 podStartE2EDuration="3.609228806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:18.608004721 +0000 UTC m=+823.551177914" watchObservedRunningTime="2026-10-30 00:52:18.609228806 +0000 UTC m=+823.552402000" Oct 30 00:52:19.018843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.018627 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:41622->10.132.0.60:15021: read: connection reset by peer" start-of-body= Oct 30 00:52:19.018843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.018690 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:41622->10.132.0.60:15021: read: connection reset by peer" Oct 30 00:52:19.105547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.102947 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:19.105547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.103015 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:19.391593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.390566 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:19.391593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.390721 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 00:52:19.391593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.390766 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 00:52:19.423863 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.423829 4905 generic.go:356] "Generic (PLEG): container finished" podID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerID="9db5876a683516c0129d0c3344125381824bca37d5cb868c9bebf078b52107be" exitCode=0 Oct 30 00:52:19.438542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.424778 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" event={"ID":"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1","Type":"ContainerDied","Data":"9db5876a683516c0129d0c3344125381824bca37d5cb868c9bebf078b52107be"} Oct 30 00:52:19.438542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.425171 4905 scope.go:122] "RemoveContainer" containerID="9db5876a683516c0129d0c3344125381824bca37d5cb868c9bebf078b52107be" Oct 30 00:52:19.459546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.446749 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:19.459546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.446851 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:19.459546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.446895 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:19.499867 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.499842 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:52:19.507172 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.507142 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 00:52:19.769000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.768633 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:19.769000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:19.768701 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:20.098280 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.097846 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:52:20.098280 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.097900 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 00:52:20.389595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.386860 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 00:52:20.389595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.386919 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 00:52:20.524439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.524396 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" event={"ID":"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1","Type":"ContainerStarted","Data":"59541bda9b9e3d0bd462ac76ed3b22a3f6a50fb4fd1bc537fa29ffdb470eb0dd"} Oct 30 00:52:20.791904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.791875 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:52:20.792580 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.792554 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:52:20.792754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:20.792726 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 00:52:21.119086 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:21.117743 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:52:21.128207 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:21.126737 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 00:52:21.398593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:21.393893 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 00:52:21.398593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:21.393964 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 00:52:21.794554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:21.792899 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:52:21.794554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:21.792976 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 00:52:22.386844 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:22.386805 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 00:52:22.387044 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:22.386858 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 00:52:22.790284 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:22.789971 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:52:22.790284 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:22.790020 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 00:52:23.386780 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:23.386750 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 00:52:23.387260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:23.387220 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 00:52:23.790069 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:23.790028 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:52:23.790273 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:23.790087 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 00:52:24.387058 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:24.387017 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 00:52:24.387579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:24.387071 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 00:52:24.790897 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:24.790862 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:52:24.791071 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:24.790926 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 00:52:25.390962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:25.390887 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:25.391470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:25.391197 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:25.391987 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:25.391962 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 00:52:25.790461 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:25.790380 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:52:25.790461 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:25.790446 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 00:52:26.594710 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.594682 4905 generic.go:356] "Generic (PLEG): container finished" podID="6392db76-3331-49b5-9bf3-055e24d0f203" containerID="04909fc8e66b995e3b0d85b10ff8e024e884703509bf2b86275a93b164d631df" exitCode=0 Oct 30 00:52:26.595097 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.594760 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" event={"ID":"6392db76-3331-49b5-9bf3-055e24d0f203","Type":"ContainerDied","Data":"04909fc8e66b995e3b0d85b10ff8e024e884703509bf2b86275a93b164d631df"} Oct 30 00:52:26.595262 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.595247 4905 scope.go:122] "RemoveContainer" containerID="04909fc8e66b995e3b0d85b10ff8e024e884703509bf2b86275a93b164d631df" Oct 30 00:52:26.596492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.596464 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:26.596575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.596548 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:26.596615 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.596583 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:52:26.794167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.794142 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:52:26.794435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.794416 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:52:26.795165 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:26.795150 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 00:52:27.603329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:27.603293 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" event={"ID":"6392db76-3331-49b5-9bf3-055e24d0f203","Type":"ContainerStarted","Data":"c27235258f8a83d4327f15f57afe2133fd5850c7e584c5cdabef56781cc24d9a"} Oct 30 00:52:28.325547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:28.323992 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:52:28.325547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:28.324109 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:52:28.325547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:28.324155 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 00:52:29.321864 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:29.321822 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:52:29.322268 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:29.321879 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 00:52:30.322180 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:30.322142 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:52:30.322607 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:30.322202 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 00:52:31.184127 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.183610 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf"] Oct 30 00:52:31.189602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.189391 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.206867 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.206836 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf"] Oct 30 00:52:31.216556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.216490 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"kube-root-ca.crt\"" Oct 30 00:52:31.216663 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.216605 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"openshift-service-ca.crt\"" Oct 30 00:52:31.219414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.219390 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-home\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.219585 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.219456 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-model-cache\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.219585 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.219505 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tmp-dir\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.219585 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.219573 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsqfr\" (UniqueName: \"kubernetes.io/projected/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-kube-api-access-zsqfr\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.219842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.219652 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tls-certs\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.219842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.219696 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-dshm\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.219993 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.219960 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"default-dockercfg-kdmj6\"" Oct 30 00:52:31.220073 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.220032 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"simulator-w6-5a344d-kserve-self-signed-certs\"" Oct 30 00:52:31.320545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.320487 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsqfr\" (UniqueName: \"kubernetes.io/projected/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-kube-api-access-zsqfr\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.320707 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.320590 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tls-certs\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.320707 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.320623 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-dshm\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.320707 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.320663 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-home\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.320824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.320714 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-model-cache\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.320824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.320761 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tmp-dir\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.321125 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.321102 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-home\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.321180 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.321122 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-model-cache\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.321217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.321186 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tmp-dir\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.321631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.321594 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:52:31.321705 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.321668 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 00:52:31.323007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.322991 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-dshm\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.323432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.323412 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tls-certs\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.359443 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.359231 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsqfr\" (UniqueName: \"kubernetes.io/projected/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-kube-api-access-zsqfr\") pod \"simulator-w6-5a344d-kserve-748fd759c4-5vwgf\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.507136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.507102 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:31.920569 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:52:31.920540 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03cbfb3c_86d3_4d4c_bf04_320b5be8ae1c.slice/crio-5b6db60c740cf70ced1f87dd0dc475d8bf2e284b136c485043fd87b6e11b846e WatchSource:0}: Error finding container 5b6db60c740cf70ced1f87dd0dc475d8bf2e284b136c485043fd87b6e11b846e: Status 404 returned error can't find the container with id 5b6db60c740cf70ced1f87dd0dc475d8bf2e284b136c485043fd87b6e11b846e Oct 30 00:52:31.920769 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:31.920750 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf"] Oct 30 00:52:32.321850 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:32.321812 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:52:32.322017 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:32.321864 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 00:52:32.654819 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:32.654729 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" event={"ID":"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c","Type":"ContainerStarted","Data":"b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37"} Oct 30 00:52:32.654819 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:32.654774 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" event={"ID":"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c","Type":"ContainerStarted","Data":"5b6db60c740cf70ced1f87dd0dc475d8bf2e284b136c485043fd87b6e11b846e"} Oct 30 00:52:32.696908 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:32.696864 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podStartSLOduration=1.696851461 podStartE2EDuration="1.696851461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:32.696114685 +0000 UTC m=+837.639287877" watchObservedRunningTime="2026-10-30 00:52:32.696851461 +0000 UTC m=+837.640024664" Oct 30 00:52:33.326480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:33.326444 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:52:33.327380 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:33.327350 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:52:33.328222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:33.328195 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:52:33.508159 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:33.508126 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:35.456455 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.456424 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz"] Oct 30 00:52:35.461299 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.461263 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.478404 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.478375 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"default-dockercfg-t4sjj\"" Oct 30 00:52:35.482807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.482784 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz"] Oct 30 00:52:35.492247 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.492223 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"kube-root-ca.crt\"" Oct 30 00:52:35.492247 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.492234 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"openshift-service-ca.crt\"" Oct 30 00:52:35.577620 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.577597 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"premium-w1-8cf33a-kserve-self-signed-certs\"" Oct 30 00:52:35.594170 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.594144 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-dshm\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.594335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.594174 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rjh\" (UniqueName: \"kubernetes.io/projected/dff3d451-bc75-41a9-8048-26cbb03cab06-kube-api-access-f6rjh\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.594335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.594270 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-tmp-dir\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.594335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.594314 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-model-cache\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.594456 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.594417 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-home\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.594456 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.594452 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff3d451-bc75-41a9-8048-26cbb03cab06-tls-certs\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.699168 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-home\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.699239 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff3d451-bc75-41a9-8048-26cbb03cab06-tls-certs\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.699335 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-dshm\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.699361 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rjh\" (UniqueName: \"kubernetes.io/projected/dff3d451-bc75-41a9-8048-26cbb03cab06-kube-api-access-f6rjh\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.699406 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-tmp-dir\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.699448 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-model-cache\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.699981 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-model-cache\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.700217 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-home\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.702543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.702015 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-tmp-dir\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.726135 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.725070 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff3d451-bc75-41a9-8048-26cbb03cab06-tls-certs\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.726135 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.725852 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rjh\" (UniqueName: \"kubernetes.io/projected/dff3d451-bc75-41a9-8048-26cbb03cab06-kube-api-access-f6rjh\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.735604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.735418 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-dshm\") pod \"premium-w1-8cf33a-kserve-5df586d94c-gl2dz\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:35.785656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:35.785612 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:36.081735 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:52:36.081706 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddff3d451_bc75_41a9_8048_26cbb03cab06.slice/crio-ce423bd09703343e69cf83c562396ad4ca7765863a9bba3af94944cb587013d0 WatchSource:0}: Error finding container ce423bd09703343e69cf83c562396ad4ca7765863a9bba3af94944cb587013d0: Status 404 returned error can't find the container with id ce423bd09703343e69cf83c562396ad4ca7765863a9bba3af94944cb587013d0 Oct 30 00:52:36.082367 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:36.082292 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz"] Oct 30 00:52:36.716209 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:36.716173 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" event={"ID":"dff3d451-bc75-41a9-8048-26cbb03cab06","Type":"ContainerStarted","Data":"ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861"} Oct 30 00:52:36.716672 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:36.716221 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" event={"ID":"dff3d451-bc75-41a9-8048-26cbb03cab06","Type":"ContainerStarted","Data":"ce423bd09703343e69cf83c562396ad4ca7765863a9bba3af94944cb587013d0"} Oct 30 00:52:36.759839 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:36.759797 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podStartSLOduration=1.7597844459999998 podStartE2EDuration="1.759784446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:36.757815177 +0000 UTC m=+841.700988369" watchObservedRunningTime="2026-10-30 00:52:36.759784446 +0000 UTC m=+841.702957694" Oct 30 00:52:36.785888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:36.785865 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:41.508468 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:41.508434 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:41.520483 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:41.520459 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:41.830789 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:41.830710 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:52:42.413909 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:52:42.413875 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:52:45.786303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:45.786232 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:45.799970 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:45.799947 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:45.859086 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:45.859051 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:52:52.533584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.533543 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9"] Oct 30 00:52:52.537698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.537674 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.558478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.558447 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9"] Oct 30 00:52:52.568937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.568915 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"openshift-service-ca.crt\"" Oct 30 00:52:52.569050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.568941 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"premium-w0-3c2596-kserve-self-signed-certs\"" Oct 30 00:52:52.569129 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.568941 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"kube-root-ca.crt\"" Oct 30 00:52:52.569129 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.569002 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"default-dockercfg-jgw45\"" Oct 30 00:52:52.613195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.613168 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-dshm\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.613195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.613198 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-model-cache\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.613361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.613218 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d62e24-c720-4e25-a170-a5740a38ee8f-tls-certs\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.613361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.613239 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-home\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.613451 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.613372 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6z7c\" (UniqueName: \"kubernetes.io/projected/09d62e24-c720-4e25-a170-a5740a38ee8f-kube-api-access-d6z7c\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.613451 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.613443 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-tmp-dir\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714419 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714383 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-tmp-dir\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714460 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-dshm\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714486 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-model-cache\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714551 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d62e24-c720-4e25-a170-a5740a38ee8f-tls-certs\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714581 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-home\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714689 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6z7c\" (UniqueName: \"kubernetes.io/projected/09d62e24-c720-4e25-a170-a5740a38ee8f-kube-api-access-d6z7c\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714863 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714827 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-tmp-dir\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.714863 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714844 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-model-cache\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.715008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.714988 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-home\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.716843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.716818 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-dshm\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.717025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.717010 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d62e24-c720-4e25-a170-a5740a38ee8f-tls-certs\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.746587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.746489 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6z7c\" (UniqueName: \"kubernetes.io/projected/09d62e24-c720-4e25-a170-a5740a38ee8f-kube-api-access-d6z7c\") pod \"premium-w0-3c2596-kserve-b8ddcb66b-xf4p9\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:52.850944 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:52.850851 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:52:53.148348 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:52:53.148317 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d62e24_c720_4e25_a170_a5740a38ee8f.slice/crio-a76a33c544541ef12546d7dab1b0cfa071f567c7b16f69b66c15c536112f9362 WatchSource:0}: Error finding container a76a33c544541ef12546d7dab1b0cfa071f567c7b16f69b66c15c536112f9362: Status 404 returned error can't find the container with id a76a33c544541ef12546d7dab1b0cfa071f567c7b16f69b66c15c536112f9362 Oct 30 00:52:53.149059 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:53.149038 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9"] Oct 30 00:52:53.901442 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:53.901406 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" event={"ID":"09d62e24-c720-4e25-a170-a5740a38ee8f","Type":"ContainerStarted","Data":"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c"} Oct 30 00:52:53.901964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:53.901456 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" event={"ID":"09d62e24-c720-4e25-a170-a5740a38ee8f","Type":"ContainerStarted","Data":"a76a33c544541ef12546d7dab1b0cfa071f567c7b16f69b66c15c536112f9362"} Oct 30 00:52:54.851790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:52:54.851753 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:53:00.744062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.743999 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podStartSLOduration=8.743980278 podStartE2EDuration="8.743980278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:53.944713024 +0000 UTC m=+858.887886215" watchObservedRunningTime="2026-10-30 00:53:00.743980278 +0000 UTC m=+865.687153472" Oct 30 00:53:00.745279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.745257 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r"] Oct 30 00:53:00.750060 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.750026 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.785730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.785699 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r"] Oct 30 00:53:00.808679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808485 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.808679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808575 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445bf3a-7f07-4bc5-a092-3eef8b985999-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.808679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808627 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.808971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808711 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.808971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808752 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.808971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808801 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.808971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808867 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zm5q\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-kube-api-access-8zm5q\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.808971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.808951 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.810036 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.809000 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910152 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zm5q\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-kube-api-access-8zm5q\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910225 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910253 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910270 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910295 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445bf3a-7f07-4bc5-a092-3eef8b985999-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910437 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910564 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910606 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.910719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910657 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.911046 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910770 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.911046 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.910932 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.911175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.911140 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.911243 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.911197 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445bf3a-7f07-4bc5-a092-3eef8b985999-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.911309 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.911286 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.912763 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.912741 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.913166 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.913148 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.933513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.933487 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:00.934224 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:00.934201 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zm5q\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-kube-api-access-8zm5q\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:01.072139 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:01.072048 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:01.384296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:01.384266 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r"] Oct 30 00:53:01.384467 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:01.384445 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8445bf3a_7f07_4bc5_a092_3eef8b985999.slice/crio-1044f63f3d42697d33fd7f3a917f778197b4fe601fea471d0904c73681346b7f WatchSource:0}: Error finding container 1044f63f3d42697d33fd7f3a917f778197b4fe601fea471d0904c73681346b7f: Status 404 returned error can't find the container with id 1044f63f3d42697d33fd7f3a917f778197b4fe601fea471d0904c73681346b7f Oct 30 00:53:01.386787 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:01.386758 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:53:01.386864 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:01.386824 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:53:01.386864 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:01.386852 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:53:01.958248 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:01.958211 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" event={"ID":"8445bf3a-7f07-4bc5-a092-3eef8b985999","Type":"ContainerStarted","Data":"2446d6b5e1ce21c5a20a13f4eecdf575b8f556d28cc7f5fc8046dc278f446641"} Oct 30 00:53:01.958663 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:01.958274 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" event={"ID":"8445bf3a-7f07-4bc5-a092-3eef8b985999","Type":"ContainerStarted","Data":"1044f63f3d42697d33fd7f3a917f778197b4fe601fea471d0904c73681346b7f"} Oct 30 00:53:02.016377 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:02.016330 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podStartSLOduration=2.01631738 podStartE2EDuration="2.01631738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:02.01444383 +0000 UTC m=+866.957617022" watchObservedRunningTime="2026-10-30 00:53:02.01631738 +0000 UTC m=+866.959490570" Oct 30 00:53:02.073197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:02.073152 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:02.075210 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:02.075181 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:53:02.075358 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:02.075231 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 00:53:02.853543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:02.853368 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:53:02.868644 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:02.868614 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:53:02.974357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:02.974322 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:53:03.073053 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:03.072746 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:53:03.073053 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:03.072796 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 00:53:04.073409 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:04.073377 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:53:04.073904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:04.073428 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 00:53:05.072580 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:05.072548 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:53:05.072761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:05.072599 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 00:53:06.072645 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:06.072608 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:53:06.073071 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:06.072673 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 00:53:07.073137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:07.073098 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:53:07.073604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:07.073160 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 00:53:08.077789 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:08.077758 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:08.078392 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:08.078208 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:08.078912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:08.078894 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 00:53:10.089198 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.089165 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6"] Oct 30 00:53:10.094583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.094559 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.126966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.126934 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"kube-root-ca.crt\"" Oct 30 00:53:10.127138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.126997 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"premium-w4-b565ec-kserve-self-signed-certs\"" Oct 30 00:53:10.127138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.127027 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"openshift-service-ca.crt\"" Oct 30 00:53:10.127138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.127052 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"default-dockercfg-7hgn4\"" Oct 30 00:53:10.131685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.131660 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6"] Oct 30 00:53:10.164568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.164539 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwzn\" (UniqueName: \"kubernetes.io/projected/b2d3906c-bd1a-4817-afc3-bb8158310757-kube-api-access-pmwzn\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.164730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.164576 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d3906c-bd1a-4817-afc3-bb8158310757-tls-certs\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.164730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.164716 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-dshm\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.164876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.164798 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-tmp-dir\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.164943 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.164915 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-home\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.164996 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.164967 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-model-cache\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.274638 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwzn\" (UniqueName: \"kubernetes.io/projected/b2d3906c-bd1a-4817-afc3-bb8158310757-kube-api-access-pmwzn\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.274758 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d3906c-bd1a-4817-afc3-bb8158310757-tls-certs\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.274899 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-dshm\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.275022 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-tmp-dir\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.275087 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-home\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.275124 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-model-cache\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.275635 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-model-cache\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.275934 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-tmp-dir\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.282540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.276164 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-home\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.284850 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.284818 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d3906c-bd1a-4817-afc3-bb8158310757-tls-certs\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.285728 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.285705 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-dshm\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.309195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.309164 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwzn\" (UniqueName: \"kubernetes.io/projected/b2d3906c-bd1a-4817-afc3-bb8158310757-kube-api-access-pmwzn\") pod \"premium-w4-b565ec-kserve-d4664b57d-q9vw6\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.410182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.410102 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:10.710205 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:10.710176 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2d3906c_bd1a_4817_afc3_bb8158310757.slice/crio-13b496ae0b957df6a9b1d3673660b8f3cca1613f9b4a025edb068f0728da7d4d WatchSource:0}: Error finding container 13b496ae0b957df6a9b1d3673660b8f3cca1613f9b4a025edb068f0728da7d4d: Status 404 returned error can't find the container with id 13b496ae0b957df6a9b1d3673660b8f3cca1613f9b4a025edb068f0728da7d4d Oct 30 00:53:10.710976 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:10.710953 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6"] Oct 30 00:53:11.017872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:11.017838 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" event={"ID":"b2d3906c-bd1a-4817-afc3-bb8158310757","Type":"ContainerStarted","Data":"12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e"} Oct 30 00:53:11.017872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:11.017877 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" event={"ID":"b2d3906c-bd1a-4817-afc3-bb8158310757","Type":"ContainerStarted","Data":"13b496ae0b957df6a9b1d3673660b8f3cca1613f9b4a025edb068f0728da7d4d"} Oct 30 00:53:11.060507 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:11.060454 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podStartSLOduration=1.060439171 podStartE2EDuration="1.060439171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:11.05984494 +0000 UTC m=+876.003018134" watchObservedRunningTime="2026-10-30 00:53:11.060439171 +0000 UTC m=+876.003612363" Oct 30 00:53:11.410622 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:11.410547 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:12.114480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.114452 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd"] Oct 30 00:53:12.119616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.119595 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.141536 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.139070 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd"] Oct 30 00:53:12.148313 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.148280 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"premium-w6-5a344d-kserve-self-signed-certs\"" Oct 30 00:53:12.239871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.239841 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-tmp-dir\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.240050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.239879 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-dshm\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.240050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.239918 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxdt\" (UniqueName: \"kubernetes.io/projected/9615f726-7b53-4e84-a623-49a37baa3af3-kube-api-access-mgxdt\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.240050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.240010 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-model-cache\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.240050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.240043 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-home\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.240264 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.240064 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9615f726-7b53-4e84-a623-49a37baa3af3-tls-certs\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.343249 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.343203 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-tmp-dir\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.344048 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.344026 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-dshm\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.344275 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.344257 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxdt\" (UniqueName: \"kubernetes.io/projected/9615f726-7b53-4e84-a623-49a37baa3af3-kube-api-access-mgxdt\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.344407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.344394 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-model-cache\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.344543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.344510 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-home\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.344664 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.344648 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9615f726-7b53-4e84-a623-49a37baa3af3-tls-certs\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.349878 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.343965 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-tmp-dir\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.354747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.354722 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-home\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.355464 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.355444 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-model-cache\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.363335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.363298 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9615f726-7b53-4e84-a623-49a37baa3af3-tls-certs\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.364034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.363993 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-dshm\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.376293 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.376209 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxdt\" (UniqueName: \"kubernetes.io/projected/9615f726-7b53-4e84-a623-49a37baa3af3-kube-api-access-mgxdt\") pod \"premium-w6-5a344d-kserve-69cb799899-xx9fd\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.434552 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:53:12.434503 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:53:12.442916 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.442886 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:12.744323 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:12.744290 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd"] Oct 30 00:53:13.032218 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.032129 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" event={"ID":"9615f726-7b53-4e84-a623-49a37baa3af3","Type":"ContainerStarted","Data":"4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d"} Oct 30 00:53:13.032218 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.032179 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" event={"ID":"9615f726-7b53-4e84-a623-49a37baa3af3","Type":"ContainerStarted","Data":"6d41e6c1ea3ddb00800a088c2c72abbc53ae41c10f322c2b8cea51732eec97c8"} Oct 30 00:53:13.073381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.073336 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podStartSLOduration=1.073323246 podStartE2EDuration="1.073323246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:13.072855477 +0000 UTC m=+878.016028674" watchObservedRunningTime="2026-10-30 00:53:13.073323246 +0000 UTC m=+878.016496437" Oct 30 00:53:13.443809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.443734 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:13.743808 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.743774 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8"] Oct 30 00:53:13.747778 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.747763 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.766869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.766846 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8"] Oct 30 00:53:13.777416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.777396 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:53:13.777594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.777498 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:53:13.777730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.777638 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:53:13.778257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.778241 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:53:13.863723 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.863689 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvn2f\" (UniqueName: \"kubernetes.io/projected/5b63afef-da01-4e70-8548-be492ba34674-kube-api-access-dvn2f\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.863912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.863897 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b63afef-da01-4e70-8548-be492ba34674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.863966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.863935 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b63afef-da01-4e70-8548-be492ba34674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.965296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.965272 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b63afef-da01-4e70-8548-be492ba34674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.965431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.965314 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b63afef-da01-4e70-8548-be492ba34674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.965431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.965405 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvn2f\" (UniqueName: \"kubernetes.io/projected/5b63afef-da01-4e70-8548-be492ba34674-kube-api-access-dvn2f\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.966041 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.966014 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b63afef-da01-4e70-8548-be492ba34674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.968066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.968049 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b63afef-da01-4e70-8548-be492ba34674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:13.986151 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:13.986126 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvn2f\" (UniqueName: \"kubernetes.io/projected/5b63afef-da01-4e70-8548-be492ba34674-kube-api-access-dvn2f\") pod \"maas-api-revoke-sub-a53ea6b0-xg5f8\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:14.062575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:14.062490 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:14.371105 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:14.371078 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b63afef_da01_4e70_8548_be492ba34674.slice/crio-94d91589b08b596d7226ca4e0e2b6a76679fe2c6757de1bc89f71484c0685a69 WatchSource:0}: Error finding container 94d91589b08b596d7226ca4e0e2b6a76679fe2c6757de1bc89f71484c0685a69: Status 404 returned error can't find the container with id 94d91589b08b596d7226ca4e0e2b6a76679fe2c6757de1bc89f71484c0685a69 Oct 30 00:53:14.371664 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:14.371642 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8"] Oct 30 00:53:15.048923 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:15.048886 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" event={"ID":"5b63afef-da01-4e70-8548-be492ba34674","Type":"ContainerStarted","Data":"94d91589b08b596d7226ca4e0e2b6a76679fe2c6757de1bc89f71484c0685a69"} Oct 30 00:53:18.146010 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:18.145965 4905 generic.go:356] "Generic (PLEG): container finished" podID="5b63afef-da01-4e70-8548-be492ba34674" containerID="cd3e9a9fc3fb4a3180d92333d3dd2f4487c7211823d6db5157848c0cc4dba0ef" exitCode=0 Oct 30 00:53:18.146720 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:18.146691 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" event={"ID":"5b63afef-da01-4e70-8548-be492ba34674","Type":"ContainerDied","Data":"cd3e9a9fc3fb4a3180d92333d3dd2f4487c7211823d6db5157848c0cc4dba0ef"} Oct 30 00:53:19.682810 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.682783 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:19.788720 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.788642 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b63afef-da01-4e70-8548-be492ba34674-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b63afef-da01-4e70-8548-be492ba34674-maas-api-openshift-ca\") pod \"5b63afef-da01-4e70-8548-be492ba34674\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " Oct 30 00:53:19.788871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.788741 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b63afef-da01-4e70-8548-be492ba34674-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b63afef-da01-4e70-8548-be492ba34674-maas-api-xks-ca\") pod \"5b63afef-da01-4e70-8548-be492ba34674\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " Oct 30 00:53:19.788871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.788777 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b63afef-da01-4e70-8548-be492ba34674-kube-api-access-dvn2f\" (UniqueName: \"kubernetes.io/projected/5b63afef-da01-4e70-8548-be492ba34674-kube-api-access-dvn2f\") pod \"5b63afef-da01-4e70-8548-be492ba34674\" (UID: \"5b63afef-da01-4e70-8548-be492ba34674\") " Oct 30 00:53:19.789077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.789053 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b63afef-da01-4e70-8548-be492ba34674-maas-api-openshift-ca" pod "5b63afef-da01-4e70-8548-be492ba34674" (UID: "5b63afef-da01-4e70-8548-be492ba34674"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:19.791287 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.791257 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b63afef-da01-4e70-8548-be492ba34674-maas-api-xks-ca" pod "5b63afef-da01-4e70-8548-be492ba34674" (UID: "5b63afef-da01-4e70-8548-be492ba34674"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:19.791413 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.791288 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b63afef-da01-4e70-8548-be492ba34674-kube-api-access-dvn2f" pod "5b63afef-da01-4e70-8548-be492ba34674" (UID: "5b63afef-da01-4e70-8548-be492ba34674"). InnerVolumeSpecName "kube-api-access-dvn2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:19.890280 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.890249 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b63afef-da01-4e70-8548-be492ba34674-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:19.890280 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.890278 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b63afef-da01-4e70-8548-be492ba34674-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:19.890280 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:19.890287 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvn2f\" (UniqueName: \"kubernetes.io/projected/5b63afef-da01-4e70-8548-be492ba34674-kube-api-access-dvn2f\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:20.222217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.222141 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" Oct 30 00:53:20.222217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.222166 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8" event={"ID":"5b63afef-da01-4e70-8548-be492ba34674","Type":"ContainerDied","Data":"94d91589b08b596d7226ca4e0e2b6a76679fe2c6757de1bc89f71484c0685a69"} Oct 30 00:53:20.222217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.222214 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94d91589b08b596d7226ca4e0e2b6a76679fe2c6757de1bc89f71484c0685a69" Oct 30 00:53:20.411242 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.411210 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:20.424115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.424093 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:20.864889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.864856 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9"] Oct 30 00:53:20.877543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.874464 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5b63afef-da01-4e70-8548-be492ba34674" containerName="revoke-keys" Oct 30 00:53:20.877752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.877733 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b63afef-da01-4e70-8548-be492ba34674" containerName="revoke-keys" Oct 30 00:53:20.878179 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.878161 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b63afef-da01-4e70-8548-be492ba34674" containerName="revoke-keys" Oct 30 00:53:20.884681 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.884656 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:20.885761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.885716 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9"] Oct 30 00:53:20.914733 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.914715 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"default-dockercfg-kbp2n\"" Oct 30 00:53:20.914978 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.914961 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"premium-w3-298ddc-kserve-self-signed-certs\"" Oct 30 00:53:20.915075 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.914979 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"kube-root-ca.crt\"" Oct 30 00:53:20.915075 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:20.915046 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"openshift-service-ca.crt\"" Oct 30 00:53:21.004113 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.004083 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-home\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.004261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.004133 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57x6r\" (UniqueName: \"kubernetes.io/projected/5f264934-e2ee-435e-898f-921f20b00f16-kube-api-access-57x6r\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.004261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.004212 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f264934-e2ee-435e-898f-921f20b00f16-tls-certs\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.004381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.004307 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-tmp-dir\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.004442 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.004424 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-dshm\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.004503 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.004469 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-model-cache\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.122613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.122122 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-home\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.122613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.122217 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57x6r\" (UniqueName: \"kubernetes.io/projected/5f264934-e2ee-435e-898f-921f20b00f16-kube-api-access-57x6r\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.122613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.122261 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f264934-e2ee-435e-898f-921f20b00f16-tls-certs\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.122613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.122307 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-tmp-dir\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.122613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.122414 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-dshm\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.122613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.122452 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-model-cache\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.123045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.122966 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-model-cache\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.130182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.123184 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-home\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.130182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.123811 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-tmp-dir\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.141559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.133838 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-dshm\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.144409 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.143790 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f264934-e2ee-435e-898f-921f20b00f16-tls-certs\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.146900 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.146876 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57x6r\" (UniqueName: \"kubernetes.io/projected/5f264934-e2ee-435e-898f-921f20b00f16-kube-api-access-57x6r\") pod \"premium-w3-298ddc-kserve-86b6b4d799-25ff9\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.200712 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.200689 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:21.240530 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.240457 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:53:21.501015 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:21.500980 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f264934_e2ee_435e_898f_921f20b00f16.slice/crio-81be80d14b9f0f765045f1a69711df07a3ff1b5ca4484238a92b85154578471a WatchSource:0}: Error finding container 81be80d14b9f0f765045f1a69711df07a3ff1b5ca4484238a92b85154578471a: Status 404 returned error can't find the container with id 81be80d14b9f0f765045f1a69711df07a3ff1b5ca4484238a92b85154578471a Oct 30 00:53:21.501382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:21.501361 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9"] Oct 30 00:53:22.238107 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:22.238064 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" event={"ID":"5f264934-e2ee-435e-898f-921f20b00f16","Type":"ContainerStarted","Data":"c432493aa78825acdbedf15ed643774c4f03fbe8a0bd6f82d92b99e1cd2db135"} Oct 30 00:53:22.238598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:22.238118 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" event={"ID":"5f264934-e2ee-435e-898f-921f20b00f16","Type":"ContainerStarted","Data":"81be80d14b9f0f765045f1a69711df07a3ff1b5ca4484238a92b85154578471a"} Oct 30 00:53:22.294611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:22.294563 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podStartSLOduration=2.294549668 podStartE2EDuration="2.294549668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:22.291188398 +0000 UTC m=+887.234361588" watchObservedRunningTime="2026-10-30 00:53:22.294549668 +0000 UTC m=+887.237722854" Oct 30 00:53:22.444694 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:22.444655 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:22.460976 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:22.460949 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:23.201045 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:23.201015 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:23.254669 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:23.254641 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:53:31.223564 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:31.222095 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:31.274389 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:31.274359 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:31.377789 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:31.377544 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:53:34.221012 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.220979 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct"] Oct 30 00:53:34.226366 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.226337 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.245035 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.245006 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct"] Oct 30 00:53:34.255095 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.255020 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:53:34.255095 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.255058 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:53:34.255095 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.255063 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:53:34.255858 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.255844 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:53:34.331664 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.331633 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.331829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.331753 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.331829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.331801 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxksl\" (UniqueName: \"kubernetes.io/projected/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-kube-api-access-xxksl\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.432921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.432888 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.433084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.433011 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.433084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.433050 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxksl\" (UniqueName: \"kubernetes.io/projected/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-kube-api-access-xxksl\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.433676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.433643 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.435578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.435557 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.453610 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.453576 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxksl\" (UniqueName: \"kubernetes.io/projected/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-kube-api-access-xxksl\") pod \"maas-api-revoke-sub-1e0d2fc5-qtgct\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.544634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.544600 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:34.718434 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.718396 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk"] Oct 30 00:53:34.723449 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.723430 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.736324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.736293 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgl75\" (UniqueName: \"kubernetes.io/projected/2c4ea89b-62f6-411e-9130-ede9e25de82c-kube-api-access-zgl75\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.736503 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.736464 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.736657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.736635 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.741904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.741880 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk"] Oct 30 00:53:34.840603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.838293 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgl75\" (UniqueName: \"kubernetes.io/projected/2c4ea89b-62f6-411e-9130-ede9e25de82c-kube-api-access-zgl75\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.840603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.838409 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.847421 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.838537 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.855929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.855890 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.866625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.866594 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.867856 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.867828 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgl75\" (UniqueName: \"kubernetes.io/projected/2c4ea89b-62f6-411e-9130-ede9e25de82c-kube-api-access-zgl75\") pod \"maas-api-revoke-sub-55d9451b-f7pmk\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:34.884906 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:34.884882 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2c8a57_acd1_478e_9dcc_abe89c15fbd8.slice/crio-58a0694600a75aaa0c148b90bb72310e12778ce70678ceca12fe902b037e0441 WatchSource:0}: Error finding container 58a0694600a75aaa0c148b90bb72310e12778ce70678ceca12fe902b037e0441: Status 404 returned error can't find the container with id 58a0694600a75aaa0c148b90bb72310e12778ce70678ceca12fe902b037e0441 Oct 30 00:53:34.885691 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:34.885668 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct"] Oct 30 00:53:35.037212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:35.037177 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:35.288754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:35.288725 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:53:35.289266 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:35.288961 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:53:35.336995 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:35.336963 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" event={"ID":"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8","Type":"ContainerStarted","Data":"322f060e9e263bc9e4642e22e3282d895cfdefbdb96b8ce2b3d93719003574c1"} Oct 30 00:53:35.337263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:35.337012 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" event={"ID":"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8","Type":"ContainerStarted","Data":"58a0694600a75aaa0c148b90bb72310e12778ce70678ceca12fe902b037e0441"} Oct 30 00:53:35.348308 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:35.348280 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4ea89b_62f6_411e_9130_ede9e25de82c.slice/crio-d57d29234fd595a1251e19234e1b2f0eeb33d01ff099801c95dd665afcb6c249 WatchSource:0}: Error finding container d57d29234fd595a1251e19234e1b2f0eeb33d01ff099801c95dd665afcb6c249: Status 404 returned error can't find the container with id d57d29234fd595a1251e19234e1b2f0eeb33d01ff099801c95dd665afcb6c249 Oct 30 00:53:35.349353 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:35.349332 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk"] Oct 30 00:53:35.380420 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:35.380376 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" podStartSLOduration=1.380362874 podStartE2EDuration="1.380362874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:35.377455715 +0000 UTC m=+900.320628909" watchObservedRunningTime="2026-10-30 00:53:35.380362874 +0000 UTC m=+900.323536065" Oct 30 00:53:36.346092 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:36.346056 4905 generic.go:356] "Generic (PLEG): container finished" podID="1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" containerID="322f060e9e263bc9e4642e22e3282d895cfdefbdb96b8ce2b3d93719003574c1" exitCode=0 Oct 30 00:53:36.346540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:36.346104 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" event={"ID":"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8","Type":"ContainerDied","Data":"322f060e9e263bc9e4642e22e3282d895cfdefbdb96b8ce2b3d93719003574c1"} Oct 30 00:53:36.347657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:36.347637 4905 generic.go:356] "Generic (PLEG): container finished" podID="2c4ea89b-62f6-411e-9130-ede9e25de82c" containerID="efb48a74cf2ccbf3abd5b65c3f6814c6a81f5c424612a76269933bc7ab0ed380" exitCode=0 Oct 30 00:53:36.347782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:36.347694 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" event={"ID":"2c4ea89b-62f6-411e-9130-ede9e25de82c","Type":"ContainerDied","Data":"efb48a74cf2ccbf3abd5b65c3f6814c6a81f5c424612a76269933bc7ab0ed380"} Oct 30 00:53:36.347782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:36.347726 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" event={"ID":"2c4ea89b-62f6-411e-9130-ede9e25de82c","Type":"ContainerStarted","Data":"d57d29234fd595a1251e19234e1b2f0eeb33d01ff099801c95dd665afcb6c249"} Oct 30 00:53:37.531284 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.531258 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:37.565426 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.565403 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:37.587762 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.587731 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-xks-ca\") pod \"2c4ea89b-62f6-411e-9130-ede9e25de82c\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " Oct 30 00:53:37.587946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.587858 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4ea89b-62f6-411e-9130-ede9e25de82c-kube-api-access-zgl75\" (UniqueName: \"kubernetes.io/projected/2c4ea89b-62f6-411e-9130-ede9e25de82c-kube-api-access-zgl75\") pod \"2c4ea89b-62f6-411e-9130-ede9e25de82c\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " Oct 30 00:53:37.588024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.587977 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-openshift-ca\") pod \"2c4ea89b-62f6-411e-9130-ede9e25de82c\" (UID: \"2c4ea89b-62f6-411e-9130-ede9e25de82c\") " Oct 30 00:53:37.588328 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.588298 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-openshift-ca" pod "2c4ea89b-62f6-411e-9130-ede9e25de82c" (UID: "2c4ea89b-62f6-411e-9130-ede9e25de82c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:37.593595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.593568 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.596654 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.596631 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-xks-ca" pod "2c4ea89b-62f6-411e-9130-ede9e25de82c" (UID: "2c4ea89b-62f6-411e-9130-ede9e25de82c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:37.596740 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.596665 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4ea89b-62f6-411e-9130-ede9e25de82c-kube-api-access-zgl75" pod "2c4ea89b-62f6-411e-9130-ede9e25de82c" (UID: "2c4ea89b-62f6-411e-9130-ede9e25de82c"). InnerVolumeSpecName "kube-api-access-zgl75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:37.694424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.694332 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-openshift-ca\") pod \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " Oct 30 00:53:37.694424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.694407 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-xks-ca\") pod \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " Oct 30 00:53:37.694695 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.694440 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-kube-api-access-xxksl\" (UniqueName: \"kubernetes.io/projected/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-kube-api-access-xxksl\") pod \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\" (UID: \"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8\") " Oct 30 00:53:37.694798 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.694749 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-openshift-ca" pod "1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" (UID: "1f2c8a57-acd1-478e-9dcc-abe89c15fbd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:37.695036 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.695012 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgl75\" (UniqueName: \"kubernetes.io/projected/2c4ea89b-62f6-411e-9130-ede9e25de82c-kube-api-access-zgl75\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.695159 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.695039 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.695159 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.695054 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea89b-62f6-411e-9130-ede9e25de82c-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.696746 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.696718 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-xks-ca" pod "1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" (UID: "1f2c8a57-acd1-478e-9dcc-abe89c15fbd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:37.696840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.696799 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-kube-api-access-xxksl" pod "1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" (UID: "1f2c8a57-acd1-478e-9dcc-abe89c15fbd8"). InnerVolumeSpecName "kube-api-access-xxksl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:37.783557 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.783502 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6"] Oct 30 00:53:37.784066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.784050 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" containerName="revoke-keys" Oct 30 00:53:37.784066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.784065 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" containerName="revoke-keys" Oct 30 00:53:37.784205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.784093 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2c4ea89b-62f6-411e-9130-ede9e25de82c" containerName="revoke-keys" Oct 30 00:53:37.784205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.784098 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4ea89b-62f6-411e-9130-ede9e25de82c" containerName="revoke-keys" Oct 30 00:53:37.784205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.784188 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4ea89b-62f6-411e-9130-ede9e25de82c" containerName="revoke-keys" Oct 30 00:53:37.784205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.784199 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" containerName="revoke-keys" Oct 30 00:53:37.787660 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.787641 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:37.797206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.796698 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.797206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.796714 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxksl\" (UniqueName: \"kubernetes.io/projected/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8-kube-api-access-xxksl\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.806860 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.806836 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6"] Oct 30 00:53:37.898602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.898559 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:37.898777 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.898657 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbzp\" (UniqueName: \"kubernetes.io/projected/21e64956-5aef-4268-bf87-35637b58ac14-kube-api-access-pzbzp\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:37.898777 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.898727 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.001876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.999925 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.001876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:37.999983 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbzp\" (UniqueName: \"kubernetes.io/projected/21e64956-5aef-4268-bf87-35637b58ac14-kube-api-access-pzbzp\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.001876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.000055 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.001876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.001105 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.011942 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.010941 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.023362 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.023330 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbzp\" (UniqueName: \"kubernetes.io/projected/21e64956-5aef-4268-bf87-35637b58ac14-kube-api-access-pzbzp\") pod \"maas-api-revoke-sub-1f826507-b8kz6\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.108175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.108140 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:38.361460 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.361372 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" event={"ID":"1f2c8a57-acd1-478e-9dcc-abe89c15fbd8","Type":"ContainerDied","Data":"58a0694600a75aaa0c148b90bb72310e12778ce70678ceca12fe902b037e0441"} Oct 30 00:53:38.361460 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.361390 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct" Oct 30 00:53:38.361460 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.361431 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a0694600a75aaa0c148b90bb72310e12778ce70678ceca12fe902b037e0441" Oct 30 00:53:38.363016 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.362981 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" event={"ID":"2c4ea89b-62f6-411e-9130-ede9e25de82c","Type":"ContainerDied","Data":"d57d29234fd595a1251e19234e1b2f0eeb33d01ff099801c95dd665afcb6c249"} Oct 30 00:53:38.363136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.363036 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57d29234fd595a1251e19234e1b2f0eeb33d01ff099801c95dd665afcb6c249" Oct 30 00:53:38.363136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.363100 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk" Oct 30 00:53:38.407113 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:38.407085 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e64956_5aef_4268_bf87_35637b58ac14.slice/crio-e1cc718f6e4abce1eee2b835add96e3b6c8270f47e58074513b77abcac8c5771 WatchSource:0}: Error finding container e1cc718f6e4abce1eee2b835add96e3b6c8270f47e58074513b77abcac8c5771: Status 404 returned error can't find the container with id e1cc718f6e4abce1eee2b835add96e3b6c8270f47e58074513b77abcac8c5771 Oct 30 00:53:38.407634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:38.407613 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6"] Oct 30 00:53:39.372344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:39.372314 4905 generic.go:356] "Generic (PLEG): container finished" podID="21e64956-5aef-4268-bf87-35637b58ac14" containerID="00bd05b48712372056a3c4679dce03905daf711ae3fa2dfc8fd6c9f1f5e68ea3" exitCode=0 Oct 30 00:53:39.372872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:39.372395 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" event={"ID":"21e64956-5aef-4268-bf87-35637b58ac14","Type":"ContainerDied","Data":"00bd05b48712372056a3c4679dce03905daf711ae3fa2dfc8fd6c9f1f5e68ea3"} Oct 30 00:53:39.372872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:39.372440 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" event={"ID":"21e64956-5aef-4268-bf87-35637b58ac14","Type":"ContainerStarted","Data":"e1cc718f6e4abce1eee2b835add96e3b6c8270f47e58074513b77abcac8c5771"} Oct 30 00:53:40.528939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.528911 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:40.651412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.651308 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-openshift-ca\") pod \"21e64956-5aef-4268-bf87-35637b58ac14\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " Oct 30 00:53:40.651612 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.651442 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e64956-5aef-4268-bf87-35637b58ac14-kube-api-access-pzbzp\" (UniqueName: \"kubernetes.io/projected/21e64956-5aef-4268-bf87-35637b58ac14-kube-api-access-pzbzp\") pod \"21e64956-5aef-4268-bf87-35637b58ac14\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " Oct 30 00:53:40.651612 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.651585 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-xks-ca\") pod \"21e64956-5aef-4268-bf87-35637b58ac14\" (UID: \"21e64956-5aef-4268-bf87-35637b58ac14\") " Oct 30 00:53:40.651769 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.651743 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-openshift-ca" pod "21e64956-5aef-4268-bf87-35637b58ac14" (UID: "21e64956-5aef-4268-bf87-35637b58ac14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:40.652034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.652009 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:40.653889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.653867 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-xks-ca" pod "21e64956-5aef-4268-bf87-35637b58ac14" (UID: "21e64956-5aef-4268-bf87-35637b58ac14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:40.654210 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.654170 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e64956-5aef-4268-bf87-35637b58ac14-kube-api-access-pzbzp" pod "21e64956-5aef-4268-bf87-35637b58ac14" (UID: "21e64956-5aef-4268-bf87-35637b58ac14"). InnerVolumeSpecName "kube-api-access-pzbzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:40.753661 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.753628 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e64956-5aef-4268-bf87-35637b58ac14-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:40.753661 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:40.753658 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbzp\" (UniqueName: \"kubernetes.io/projected/21e64956-5aef-4268-bf87-35637b58ac14-kube-api-access-pzbzp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:41.386564 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:41.386511 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" Oct 30 00:53:41.386564 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:41.386545 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6" event={"ID":"21e64956-5aef-4268-bf87-35637b58ac14","Type":"ContainerDied","Data":"e1cc718f6e4abce1eee2b835add96e3b6c8270f47e58074513b77abcac8c5771"} Oct 30 00:53:41.386819 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:41.386630 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1cc718f6e4abce1eee2b835add96e3b6c8270f47e58074513b77abcac8c5771" Oct 30 00:53:42.593551 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:53:42.577004 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:53:45.682891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.682854 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg"] Oct 30 00:53:45.683655 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.683633 4905 cpu_manager.go:414] "containerMap: removing container" podUID="21e64956-5aef-4268-bf87-35637b58ac14" containerName="revoke-keys" Oct 30 00:53:45.683655 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.683655 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e64956-5aef-4268-bf87-35637b58ac14" containerName="revoke-keys" Oct 30 00:53:45.683848 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.683834 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e64956-5aef-4268-bf87-35637b58ac14" containerName="revoke-keys" Oct 30 00:53:45.688352 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.688330 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.711482 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.711456 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:53:45.725448 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.725425 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:53:45.725646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.725488 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:53:45.725646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.725603 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:53:45.728923 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.728902 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg"] Oct 30 00:53:45.794364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.794323 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.794563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.794508 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.794636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.794587 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxlp7\" (UniqueName: \"kubernetes.io/projected/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-kube-api-access-fxlp7\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.895582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.895549 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.895775 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.895617 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxlp7\" (UniqueName: \"kubernetes.io/projected/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-kube-api-access-fxlp7\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.895775 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.895759 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.896229 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.896208 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.898302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.898275 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:45.916228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:45.916198 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxlp7\" (UniqueName: \"kubernetes.io/projected/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-kube-api-access-fxlp7\") pod \"maas-api-revoke-sub-0981c700-p2pkg\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:46.001959 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:46.001928 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:46.319759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:46.319731 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg"] Oct 30 00:53:46.322255 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:46.322230 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6184d370_9192_4f9c_bbb0_1ee3c8c0c2eb.slice/crio-adb5eebb51584f4a2c83c0518238661757ab6ef389166835dc77a7ae04c13572 WatchSource:0}: Error finding container adb5eebb51584f4a2c83c0518238661757ab6ef389166835dc77a7ae04c13572: Status 404 returned error can't find the container with id adb5eebb51584f4a2c83c0518238661757ab6ef389166835dc77a7ae04c13572 Oct 30 00:53:46.465374 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:46.465313 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" event={"ID":"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb","Type":"ContainerStarted","Data":"adb5eebb51584f4a2c83c0518238661757ab6ef389166835dc77a7ae04c13572"} Oct 30 00:53:47.473819 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:47.473782 4905 generic.go:356] "Generic (PLEG): container finished" podID="6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" containerID="c8e087981e44614c3987feff4512fe4458a4a9f108bc1391d76038a455a58630" exitCode=0 Oct 30 00:53:47.474252 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:47.473857 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" event={"ID":"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb","Type":"ContainerDied","Data":"c8e087981e44614c3987feff4512fe4458a4a9f108bc1391d76038a455a58630"} Oct 30 00:53:48.136260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.136223 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj"] Oct 30 00:53:48.140977 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.140957 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.159944 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.159901 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj"] Oct 30 00:53:48.223763 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.223736 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.223927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.223773 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzz4\" (UniqueName: \"kubernetes.io/projected/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-kube-api-access-dvzz4\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.223927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.223798 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.324772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.324741 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.324772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.324775 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzz4\" (UniqueName: \"kubernetes.io/projected/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-kube-api-access-dvzz4\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.324980 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.324799 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.325436 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.325409 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.327417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.327388 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.345033 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.345009 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzz4\" (UniqueName: \"kubernetes.io/projected/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-kube-api-access-dvzz4\") pod \"maas-api-revoke-sub-2485f24b-tqmxj\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.456616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.456513 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:48.521267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.521236 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh"] Oct 30 00:53:48.533299 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.532759 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.553563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.553109 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh"] Oct 30 00:53:48.630938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.630897 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5gv\" (UniqueName: \"kubernetes.io/projected/4d6d53c3-5657-4926-98c8-033129688a95-kube-api-access-pm5gv\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.631120 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.630950 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.631227 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.631198 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.665114 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.665093 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:48.732486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.732454 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-kube-api-access-fxlp7\" (UniqueName: \"kubernetes.io/projected/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-kube-api-access-fxlp7\") pod \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " Oct 30 00:53:48.732686 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.732568 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-xks-ca\") pod \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " Oct 30 00:53:48.732686 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.732651 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-openshift-ca\") pod \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\" (UID: \"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb\") " Oct 30 00:53:48.732957 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.732937 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.733055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.733030 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5gv\" (UniqueName: \"kubernetes.io/projected/4d6d53c3-5657-4926-98c8-033129688a95-kube-api-access-pm5gv\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.733122 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.733090 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.733981 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.733949 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-openshift-ca" pod "6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" (UID: "6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:48.734751 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.734725 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.736117 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.736092 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.736215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.736116 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-xks-ca" pod "6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" (UID: "6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:48.736748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.736718 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-kube-api-access-fxlp7" pod "6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" (UID: "6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb"). InnerVolumeSpecName "kube-api-access-fxlp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:48.753964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.753943 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5gv\" (UniqueName: \"kubernetes.io/projected/4d6d53c3-5657-4926-98c8-033129688a95-kube-api-access-pm5gv\") pod \"maas-api-revoke-sub-51dc1479-99xkh\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:48.758197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.758175 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj"] Oct 30 00:53:48.762399 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:48.762374 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0480cf_7b60_4aa2_9536_5f541ad3aa1e.slice/crio-09ba67b88cab21818b519de7c23eeaa2f7a66bb07f9aeddce2190b74f477e0f9 WatchSource:0}: Error finding container 09ba67b88cab21818b519de7c23eeaa2f7a66bb07f9aeddce2190b74f477e0f9: Status 404 returned error can't find the container with id 09ba67b88cab21818b519de7c23eeaa2f7a66bb07f9aeddce2190b74f477e0f9 Oct 30 00:53:48.834887 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.834856 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.834887 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.834887 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.835056 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.834898 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxlp7\" (UniqueName: \"kubernetes.io/projected/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb-kube-api-access-fxlp7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.853252 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:48.853223 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:49.147407 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:49.147379 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6d53c3_5657_4926_98c8_033129688a95.slice/crio-7e59a40f19b41e8a53449287e4f4269895fc6834da1c02207f306b5ef76434c0 WatchSource:0}: Error finding container 7e59a40f19b41e8a53449287e4f4269895fc6834da1c02207f306b5ef76434c0: Status 404 returned error can't find the container with id 7e59a40f19b41e8a53449287e4f4269895fc6834da1c02207f306b5ef76434c0 Oct 30 00:53:49.148801 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.148776 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh"] Oct 30 00:53:49.490853 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.490816 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" event={"ID":"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e","Type":"ContainerStarted","Data":"2305d6fe776b97adeefd4b4daa7a156186448454a07dbb47330dfa52e22b0332"} Oct 30 00:53:49.491018 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.490876 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" event={"ID":"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e","Type":"ContainerStarted","Data":"09ba67b88cab21818b519de7c23eeaa2f7a66bb07f9aeddce2190b74f477e0f9"} Oct 30 00:53:49.492192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.492159 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" event={"ID":"4d6d53c3-5657-4926-98c8-033129688a95","Type":"ContainerStarted","Data":"7e59a40f19b41e8a53449287e4f4269895fc6834da1c02207f306b5ef76434c0"} Oct 30 00:53:49.493433 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.493398 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" event={"ID":"6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb","Type":"ContainerDied","Data":"adb5eebb51584f4a2c83c0518238661757ab6ef389166835dc77a7ae04c13572"} Oct 30 00:53:49.493433 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.493415 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg" Oct 30 00:53:49.493600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.493439 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb5eebb51584f4a2c83c0518238661757ab6ef389166835dc77a7ae04c13572" Oct 30 00:53:49.537171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:49.536717 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" podStartSLOduration=1.536697957 podStartE2EDuration="1.536697957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:49.534139522 +0000 UTC m=+914.477312717" watchObservedRunningTime="2026-10-30 00:53:49.536697957 +0000 UTC m=+914.479871149" Oct 30 00:53:50.501211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:50.501177 4905 generic.go:356] "Generic (PLEG): container finished" podID="4d6d53c3-5657-4926-98c8-033129688a95" containerID="a2fdf01e6e94b9a2082891b8f2bffae08715a23e24ba1ddcd5f1261b53ecfd47" exitCode=0 Oct 30 00:53:50.501378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:50.501229 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" event={"ID":"4d6d53c3-5657-4926-98c8-033129688a95","Type":"ContainerDied","Data":"a2fdf01e6e94b9a2082891b8f2bffae08715a23e24ba1ddcd5f1261b53ecfd47"} Oct 30 00:53:50.502831 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:50.502811 4905 generic.go:356] "Generic (PLEG): container finished" podID="6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" containerID="2305d6fe776b97adeefd4b4daa7a156186448454a07dbb47330dfa52e22b0332" exitCode=0 Oct 30 00:53:50.502957 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:50.502860 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" event={"ID":"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e","Type":"ContainerDied","Data":"2305d6fe776b97adeefd4b4daa7a156186448454a07dbb47330dfa52e22b0332"} Oct 30 00:53:51.723427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.723391 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:51.726276 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.726254 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:51.876362 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876279 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-openshift-ca\") pod \"4d6d53c3-5657-4926-98c8-033129688a95\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " Oct 30 00:53:51.876362 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876323 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6d53c3-5657-4926-98c8-033129688a95-kube-api-access-pm5gv\" (UniqueName: \"kubernetes.io/projected/4d6d53c3-5657-4926-98c8-033129688a95-kube-api-access-pm5gv\") pod \"4d6d53c3-5657-4926-98c8-033129688a95\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " Oct 30 00:53:51.876605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876388 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-xks-ca\") pod \"4d6d53c3-5657-4926-98c8-033129688a95\" (UID: \"4d6d53c3-5657-4926-98c8-033129688a95\") " Oct 30 00:53:51.876605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876449 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-xks-ca\") pod \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " Oct 30 00:53:51.876605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876503 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-openshift-ca\") pod \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " Oct 30 00:53:51.876752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876655 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-kube-api-access-dvzz4\" (UniqueName: \"kubernetes.io/projected/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-kube-api-access-dvzz4\") pod \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\" (UID: \"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e\") " Oct 30 00:53:51.876752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876685 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-openshift-ca" pod "4d6d53c3-5657-4926-98c8-033129688a95" (UID: "4d6d53c3-5657-4926-98c8-033129688a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:51.876990 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876956 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-openshift-ca" pod "6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" (UID: "6a0480cf-7b60-4aa2-9536-5f541ad3aa1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:51.877117 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.876976 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:51.878836 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.878797 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-kube-api-access-dvzz4" pod "6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" (UID: "6a0480cf-7b60-4aa2-9536-5f541ad3aa1e"). InnerVolumeSpecName "kube-api-access-dvzz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:51.878836 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.878810 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-xks-ca" pod "4d6d53c3-5657-4926-98c8-033129688a95" (UID: "4d6d53c3-5657-4926-98c8-033129688a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:51.879282 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.879261 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-xks-ca" pod "6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" (UID: "6a0480cf-7b60-4aa2-9536-5f541ad3aa1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:51.879351 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.879333 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6d53c3-5657-4926-98c8-033129688a95-kube-api-access-pm5gv" pod "4d6d53c3-5657-4926-98c8-033129688a95" (UID: "4d6d53c3-5657-4926-98c8-033129688a95"). InnerVolumeSpecName "kube-api-access-pm5gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:51.978719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.978685 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzz4\" (UniqueName: \"kubernetes.io/projected/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-kube-api-access-dvzz4\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:51.978719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.978715 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5gv\" (UniqueName: \"kubernetes.io/projected/4d6d53c3-5657-4926-98c8-033129688a95-kube-api-access-pm5gv\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:51.978719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.978725 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6d53c3-5657-4926-98c8-033129688a95-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:51.978963 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.978734 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:51.978963 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:51.978743 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.520578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:52.520548 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" Oct 30 00:53:52.520578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:52.520557 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj" event={"ID":"6a0480cf-7b60-4aa2-9536-5f541ad3aa1e","Type":"ContainerDied","Data":"09ba67b88cab21818b519de7c23eeaa2f7a66bb07f9aeddce2190b74f477e0f9"} Oct 30 00:53:52.520800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:52.520607 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09ba67b88cab21818b519de7c23eeaa2f7a66bb07f9aeddce2190b74f477e0f9" Oct 30 00:53:52.522074 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:52.522054 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" Oct 30 00:53:52.522200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:52.522054 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh" event={"ID":"4d6d53c3-5657-4926-98c8-033129688a95","Type":"ContainerDied","Data":"7e59a40f19b41e8a53449287e4f4269895fc6834da1c02207f306b5ef76434c0"} Oct 30 00:53:52.522200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:52.522190 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e59a40f19b41e8a53449287e4f4269895fc6834da1c02207f306b5ef76434c0" Oct 30 00:53:55.242132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.242097 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l"] Oct 30 00:53:55.242975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.242956 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" containerName="revoke-keys" Oct 30 00:53:55.243034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.242978 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" containerName="revoke-keys" Oct 30 00:53:55.243034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.243003 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4d6d53c3-5657-4926-98c8-033129688a95" containerName="revoke-keys" Oct 30 00:53:55.243034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.243010 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6d53c3-5657-4926-98c8-033129688a95" containerName="revoke-keys" Oct 30 00:53:55.243034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.243031 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" containerName="revoke-keys" Oct 30 00:53:55.243161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.243039 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" containerName="revoke-keys" Oct 30 00:53:55.243161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.243151 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" containerName="revoke-keys" Oct 30 00:53:55.243217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.243161 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6d53c3-5657-4926-98c8-033129688a95" containerName="revoke-keys" Oct 30 00:53:55.243217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.243172 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" containerName="revoke-keys" Oct 30 00:53:55.248462 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.248435 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.275076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.275041 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:53:55.275279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.275186 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:53:55.275279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.275243 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:53:55.277837 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.277803 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:53:55.280939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.280907 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l"] Oct 30 00:53:55.380094 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.380064 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6clc\" (UniqueName: \"kubernetes.io/projected/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-kube-api-access-h6clc\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.380270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.380115 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.380326 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.380307 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.482131 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.482039 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6clc\" (UniqueName: \"kubernetes.io/projected/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-kube-api-access-h6clc\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.482131 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.482121 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.482383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.482297 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.483143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.483080 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.485490 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.485471 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.503285 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.503236 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6clc\" (UniqueName: \"kubernetes.io/projected/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-kube-api-access-h6clc\") pod \"maas-api-revoke-sub-4fd373ed-tkl6l\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.583707 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.583678 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:55.884491 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:55.884463 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9df9242_a84a_4fa1_b23d_d86b7a4f209e.slice/crio-632dc85828ad21b43d9b16413d9e15401abdb5f7247be12d7cfbd3fcc4cb7b0b WatchSource:0}: Error finding container 632dc85828ad21b43d9b16413d9e15401abdb5f7247be12d7cfbd3fcc4cb7b0b: Status 404 returned error can't find the container with id 632dc85828ad21b43d9b16413d9e15401abdb5f7247be12d7cfbd3fcc4cb7b0b Oct 30 00:53:55.885320 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:55.885260 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l"] Oct 30 00:53:56.595804 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:56.595725 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" event={"ID":"c9df9242-a84a-4fa1-b23d-d86b7a4f209e","Type":"ContainerStarted","Data":"e6bdfe2cebded9c46ed12836d4b271311e3832908524a8867f69980fdef508f1"} Oct 30 00:53:56.595804 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:56.595779 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" event={"ID":"c9df9242-a84a-4fa1-b23d-d86b7a4f209e","Type":"ContainerStarted","Data":"632dc85828ad21b43d9b16413d9e15401abdb5f7247be12d7cfbd3fcc4cb7b0b"} Oct 30 00:53:56.637729 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:56.637685 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" podStartSLOduration=1.637669735 podStartE2EDuration="1.637669735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:56.635400859 +0000 UTC m=+921.578574052" watchObservedRunningTime="2026-10-30 00:53:56.637669735 +0000 UTC m=+921.580842926" Oct 30 00:53:57.410796 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.410767 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7"] Oct 30 00:53:57.414802 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.414780 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.415802 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.415782 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.415888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.415829 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.415888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.415849 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6fp\" (UniqueName: \"kubernetes.io/projected/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-kube-api-access-4c6fp\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.434316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.434292 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7"] Oct 30 00:53:57.507407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.507374 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w"] Oct 30 00:53:57.511632 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.511613 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.516609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.516586 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.516730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.516617 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6fp\" (UniqueName: \"kubernetes.io/projected/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-kube-api-access-4c6fp\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.516730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.516652 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.516845 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.516794 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.516894 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.516858 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsdz\" (UniqueName: \"kubernetes.io/projected/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-kube-api-access-gmsdz\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.516996 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.516978 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.517416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.517392 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.519730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.519711 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.531467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.531444 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w"] Oct 30 00:53:57.536679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.536660 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6fp\" (UniqueName: \"kubernetes.io/projected/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-kube-api-access-4c6fp\") pod \"maas-api-revoke-sub-be7f7d05-6b7s7\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.617765 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.617732 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.618340 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.617837 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.618340 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.617894 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsdz\" (UniqueName: \"kubernetes.io/projected/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-kube-api-access-gmsdz\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.650379 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.640441 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.651284 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.649184 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.652573 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.651613 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsdz\" (UniqueName: \"kubernetes.io/projected/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-kube-api-access-gmsdz\") pod \"maas-api-revoke-sub-08d011ba-5qp4w\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:57.654604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.654574 4905 generic.go:356] "Generic (PLEG): container finished" podID="c9df9242-a84a-4fa1-b23d-d86b7a4f209e" containerID="e6bdfe2cebded9c46ed12836d4b271311e3832908524a8867f69980fdef508f1" exitCode=0 Oct 30 00:53:57.654713 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.654620 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" event={"ID":"c9df9242-a84a-4fa1-b23d-d86b7a4f209e","Type":"ContainerDied","Data":"e6bdfe2cebded9c46ed12836d4b271311e3832908524a8867f69980fdef508f1"} Oct 30 00:53:57.727717 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.727652 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:53:57.839551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:57.838442 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:53:58.034903 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:58.033117 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ea385e_402b_4ba5_8434_2e7cdb47c3f1.slice/crio-96314ca13c3b6e05f2b7f1d3784907a69859f88f059877296a0bc0e88a7dd73c WatchSource:0}: Error finding container 96314ca13c3b6e05f2b7f1d3784907a69859f88f059877296a0bc0e88a7dd73c: Status 404 returned error can't find the container with id 96314ca13c3b6e05f2b7f1d3784907a69859f88f059877296a0bc0e88a7dd73c Oct 30 00:53:58.034903 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.034507 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7"] Oct 30 00:53:58.161445 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:53:58.161416 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f08ddaf_ce67_44a7_8d8d_51e95e0e5c0c.slice/crio-0841e5abfb1bec7af54f8225638bef62c9572984005799a8b7f849289c8c47e9 WatchSource:0}: Error finding container 0841e5abfb1bec7af54f8225638bef62c9572984005799a8b7f849289c8c47e9: Status 404 returned error can't find the container with id 0841e5abfb1bec7af54f8225638bef62c9572984005799a8b7f849289c8c47e9 Oct 30 00:53:58.162782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.162758 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w"] Oct 30 00:53:58.665154 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.665053 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" event={"ID":"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1","Type":"ContainerStarted","Data":"e0abab8fe817b126192ce56a5dc1f59f9ae2047e03230e0cca454af12c53fbf5"} Oct 30 00:53:58.665154 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.665121 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" event={"ID":"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1","Type":"ContainerStarted","Data":"96314ca13c3b6e05f2b7f1d3784907a69859f88f059877296a0bc0e88a7dd73c"} Oct 30 00:53:58.667007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.666977 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" event={"ID":"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c","Type":"ContainerStarted","Data":"bdd80bcca47dd5928d4a97cb7c4d1362722fbba521bf51f4934ad6b50097b2dc"} Oct 30 00:53:58.667142 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.667031 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" event={"ID":"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c","Type":"ContainerStarted","Data":"0841e5abfb1bec7af54f8225638bef62c9572984005799a8b7f849289c8c47e9"} Oct 30 00:53:58.709156 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.709095 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" podStartSLOduration=1.709079208 podStartE2EDuration="1.709079208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:58.706016224 +0000 UTC m=+923.649189415" watchObservedRunningTime="2026-10-30 00:53:58.709079208 +0000 UTC m=+923.652252399" Oct 30 00:53:58.749378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.749152 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" podStartSLOduration=1.7491349870000001 podStartE2EDuration="1.749134987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:58.748780658 +0000 UTC m=+923.691953853" watchObservedRunningTime="2026-10-30 00:53:58.749134987 +0000 UTC m=+923.692308180" Oct 30 00:53:58.838940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.838914 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:58.936760 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.936689 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-xks-ca\") pod \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " Oct 30 00:53:58.936760 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.936751 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-openshift-ca\") pod \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " Oct 30 00:53:58.936974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.936867 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-kube-api-access-h6clc\" (UniqueName: \"kubernetes.io/projected/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-kube-api-access-h6clc\") pod \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\" (UID: \"c9df9242-a84a-4fa1-b23d-d86b7a4f209e\") " Oct 30 00:53:58.937078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.937054 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-openshift-ca" pod "c9df9242-a84a-4fa1-b23d-d86b7a4f209e" (UID: "c9df9242-a84a-4fa1-b23d-d86b7a4f209e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:58.937324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.937297 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:58.939001 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.938975 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-xks-ca" pod "c9df9242-a84a-4fa1-b23d-d86b7a4f209e" (UID: "c9df9242-a84a-4fa1-b23d-d86b7a4f209e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:58.939001 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:58.938992 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-kube-api-access-h6clc" pod "c9df9242-a84a-4fa1-b23d-d86b7a4f209e" (UID: "c9df9242-a84a-4fa1-b23d-d86b7a4f209e"). InnerVolumeSpecName "kube-api-access-h6clc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:59.038653 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.038620 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6clc\" (UniqueName: \"kubernetes.io/projected/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-kube-api-access-h6clc\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:59.038776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.038657 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df9242-a84a-4fa1-b23d-d86b7a4f209e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:53:59.675432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.675401 4905 generic.go:356] "Generic (PLEG): container finished" podID="e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" containerID="e0abab8fe817b126192ce56a5dc1f59f9ae2047e03230e0cca454af12c53fbf5" exitCode=0 Oct 30 00:53:59.675894 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.675493 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" event={"ID":"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1","Type":"ContainerDied","Data":"e0abab8fe817b126192ce56a5dc1f59f9ae2047e03230e0cca454af12c53fbf5"} Oct 30 00:53:59.676994 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.676974 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" Oct 30 00:53:59.677469 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.677448 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l" event={"ID":"c9df9242-a84a-4fa1-b23d-d86b7a4f209e","Type":"ContainerDied","Data":"632dc85828ad21b43d9b16413d9e15401abdb5f7247be12d7cfbd3fcc4cb7b0b"} Oct 30 00:53:59.677611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.677488 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632dc85828ad21b43d9b16413d9e15401abdb5f7247be12d7cfbd3fcc4cb7b0b" Oct 30 00:53:59.679396 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.679371 4905 generic.go:356] "Generic (PLEG): container finished" podID="8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" containerID="bdd80bcca47dd5928d4a97cb7c4d1362722fbba521bf51f4934ad6b50097b2dc" exitCode=0 Oct 30 00:53:59.679535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:53:59.679447 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" event={"ID":"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c","Type":"ContainerDied","Data":"bdd80bcca47dd5928d4a97cb7c4d1362722fbba521bf51f4934ad6b50097b2dc"} Oct 30 00:54:00.883782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.883751 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:54:00.896982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.896951 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz"] Oct 30 00:54:00.897753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.897731 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" containerName="revoke-keys" Oct 30 00:54:00.897753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.897756 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" containerName="revoke-keys" Oct 30 00:54:00.898151 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.897801 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c9df9242-a84a-4fa1-b23d-d86b7a4f209e" containerName="revoke-keys" Oct 30 00:54:00.898151 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.897811 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9df9242-a84a-4fa1-b23d-d86b7a4f209e" containerName="revoke-keys" Oct 30 00:54:00.898151 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.897975 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" containerName="revoke-keys" Oct 30 00:54:00.898151 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.897990 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9df9242-a84a-4fa1-b23d-d86b7a4f209e" containerName="revoke-keys" Oct 30 00:54:00.901822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.901801 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.917656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.917634 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz"] Oct 30 00:54:00.950274 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.949020 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:54:00.979749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.979717 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-xks-ca\") pod \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " Oct 30 00:54:00.979898 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.979786 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-openshift-ca\") pod \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " Oct 30 00:54:00.979994 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.979936 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-kube-api-access-gmsdz\" (UniqueName: \"kubernetes.io/projected/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-kube-api-access-gmsdz\") pod \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " Oct 30 00:54:00.980064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980046 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-openshift-ca\") pod \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " Oct 30 00:54:00.980382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980347 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-openshift-ca" pod "e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" (UID: "e5ea385e-402b-4ba5-8434-2e7cdb47c3f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:00.980382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980369 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-openshift-ca" pod "8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" (UID: "8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:00.980574 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980400 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-xks-ca\") pod \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\" (UID: \"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c\") " Oct 30 00:54:00.980574 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980441 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-kube-api-access-4c6fp\" (UniqueName: \"kubernetes.io/projected/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-kube-api-access-4c6fp\") pod \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\" (UID: \"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1\") " Oct 30 00:54:00.980715 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980693 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbz8\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-kube-api-access-vgbz8\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.980811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980783 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.980932 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.980831 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.981025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981002 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.981085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981044 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.981228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981209 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106de83-4886-4b14-815a-6e8fb28d9621-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.981597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981259 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106de83-4886-4b14-815a-6e8fb28d9621-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.981597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981304 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.981597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981340 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:00.981597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981503 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:00.981597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.981535 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:00.983433 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.983399 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-xks-ca" pod "e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" (UID: "e5ea385e-402b-4ba5-8434-2e7cdb47c3f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:00.983581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.983557 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-xks-ca" pod "8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" (UID: "8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:00.985029 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.985003 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-kube-api-access-gmsdz" pod "8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" (UID: "8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c"). InnerVolumeSpecName "kube-api-access-gmsdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:00.987085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:00.987065 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-kube-api-access-4c6fp" pod "e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" (UID: "e5ea385e-402b-4ba5-8434-2e7cdb47c3f1"). InnerVolumeSpecName "kube-api-access-4c6fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:01.084112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084072 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbz8\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-kube-api-access-vgbz8\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084143 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084178 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084261 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084393 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084291 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084387 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106de83-4886-4b14-815a-6e8fb28d9621-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084433 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106de83-4886-4b14-815a-6e8fb28d9621-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084497 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084478 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084512 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.084683 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084670 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:01.084735 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084691 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmsdz\" (UniqueName: \"kubernetes.io/projected/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-kube-api-access-gmsdz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:01.084735 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084707 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:01.084735 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.084721 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6fp\" (UniqueName: \"kubernetes.io/projected/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1-kube-api-access-4c6fp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:01.085113 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.085093 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.085686 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.085658 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.086036 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.086012 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.086414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.086280 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106de83-4886-4b14-815a-6e8fb28d9621-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.086637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.086557 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.088715 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.088684 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106de83-4886-4b14-815a-6e8fb28d9621-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.089142 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.089120 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.106060 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.106014 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbz8\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-kube-api-access-vgbz8\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.106714 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.106690 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.264728 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.264699 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:01.577588 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:54:01.577560 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1106de83_4886_4b14_815a_6e8fb28d9621.slice/crio-96d0f770c416e8bcd24b572dc542df2acce4df738c9d23e9cadf6292304d6804 WatchSource:0}: Error finding container 96d0f770c416e8bcd24b572dc542df2acce4df738c9d23e9cadf6292304d6804: Status 404 returned error can't find the container with id 96d0f770c416e8bcd24b572dc542df2acce4df738c9d23e9cadf6292304d6804 Oct 30 00:54:01.578504 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.578479 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz"] Oct 30 00:54:01.580146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.580115 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:01.580270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.580190 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:01.580270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.580234 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:01.695871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.695769 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" event={"ID":"1106de83-4886-4b14-815a-6e8fb28d9621","Type":"ContainerStarted","Data":"d2fcd78cc64d7106d9603657fb66163aabd6c179a97144bf049db278b1b5dfdc"} Oct 30 00:54:01.695871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.695823 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" event={"ID":"1106de83-4886-4b14-815a-6e8fb28d9621","Type":"ContainerStarted","Data":"96d0f770c416e8bcd24b572dc542df2acce4df738c9d23e9cadf6292304d6804"} Oct 30 00:54:01.697359 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.697323 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" event={"ID":"e5ea385e-402b-4ba5-8434-2e7cdb47c3f1","Type":"ContainerDied","Data":"96314ca13c3b6e05f2b7f1d3784907a69859f88f059877296a0bc0e88a7dd73c"} Oct 30 00:54:01.697491 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.697378 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96314ca13c3b6e05f2b7f1d3784907a69859f88f059877296a0bc0e88a7dd73c" Oct 30 00:54:01.697491 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.697333 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7" Oct 30 00:54:01.702206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.702174 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" event={"ID":"8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c","Type":"ContainerDied","Data":"0841e5abfb1bec7af54f8225638bef62c9572984005799a8b7f849289c8c47e9"} Oct 30 00:54:01.702206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.702197 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w" Oct 30 00:54:01.702370 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.702261 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0841e5abfb1bec7af54f8225638bef62c9572984005799a8b7f849289c8c47e9" Oct 30 00:54:01.755831 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:01.755782 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podStartSLOduration=1.755768916 podStartE2EDuration="1.755768916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:01.743796993 +0000 UTC m=+926.686970185" watchObservedRunningTime="2026-10-30 00:54:01.755768916 +0000 UTC m=+926.698942107" Oct 30 00:54:02.211296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.211265 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk"] Oct 30 00:54:02.211811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.211797 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" containerName="revoke-keys" Oct 30 00:54:02.211869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.211812 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" containerName="revoke-keys" Oct 30 00:54:02.211912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.211902 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" containerName="revoke-keys" Oct 30 00:54:02.215318 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.215297 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.239444 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.239417 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:54:02.239779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.239755 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:54:02.246249 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.246216 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk"] Oct 30 00:54:02.269970 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.269663 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:02.271245 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.271220 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:54:02.271409 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.271388 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:54:02.301063 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.301040 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.301189 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.301155 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.301303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.301285 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:54:02.301373 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.301357 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lkh\" (UniqueName: \"kubernetes.io/projected/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-kube-api-access-r2lkh\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.301413 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.301365 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:54:02.402854 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.402817 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lkh\" (UniqueName: \"kubernetes.io/projected/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-kube-api-access-r2lkh\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.403008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.402865 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.403008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.402932 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.403723 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.403698 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.405662 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.405637 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.422939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.422914 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lkh\" (UniqueName: \"kubernetes.io/projected/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-kube-api-access-r2lkh\") pod \"maas-api-revoke-sub-d7f48a62-znvdk\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.541748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.541711 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:02.862572 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:02.862507 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk"] Oct 30 00:54:03.266037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:03.266010 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:54:03.266466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:03.266066 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:54:03.718213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:03.718124 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" event={"ID":"e3ea931d-3fd7-4c23-994b-902b8cedfe2a","Type":"ContainerStarted","Data":"6217323f3155c5e8361331e091a3ca9b6e38299b009fc7d753853aac96a23e95"} Oct 30 00:54:03.718213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:03.718183 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" event={"ID":"e3ea931d-3fd7-4c23-994b-902b8cedfe2a","Type":"ContainerStarted","Data":"125669ba23ba7cd7931aa01cd19d4ae3fa8841dd461b6feaf8751a2099eff942"} Oct 30 00:54:03.765884 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:03.765817 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" podStartSLOduration=1.765798821 podStartE2EDuration="1.765798821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:03.765557318 +0000 UTC m=+928.708730524" watchObservedRunningTime="2026-10-30 00:54:03.765798821 +0000 UTC m=+928.708972015" Oct 30 00:54:04.265792 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:04.265755 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:54:04.265951 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:04.265815 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:54:04.758409 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:04.758370 4905 generic.go:356] "Generic (PLEG): container finished" podID="e3ea931d-3fd7-4c23-994b-902b8cedfe2a" containerID="6217323f3155c5e8361331e091a3ca9b6e38299b009fc7d753853aac96a23e95" exitCode=0 Oct 30 00:54:04.758891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:04.758453 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" event={"ID":"e3ea931d-3fd7-4c23-994b-902b8cedfe2a","Type":"ContainerDied","Data":"6217323f3155c5e8361331e091a3ca9b6e38299b009fc7d753853aac96a23e95"} Oct 30 00:54:05.265681 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:05.265645 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:54:05.266094 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:05.266066 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:54:06.002809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.002784 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:06.081116 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.081028 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-xks-ca\") pod \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " Oct 30 00:54:06.081281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.081114 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-openshift-ca\") pod \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " Oct 30 00:54:06.081281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.081210 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-kube-api-access-r2lkh\" (UniqueName: \"kubernetes.io/projected/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-kube-api-access-r2lkh\") pod \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\" (UID: \"e3ea931d-3fd7-4c23-994b-902b8cedfe2a\") " Oct 30 00:54:06.081778 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.081744 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-openshift-ca" pod "e3ea931d-3fd7-4c23-994b-902b8cedfe2a" (UID: "e3ea931d-3fd7-4c23-994b-902b8cedfe2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:06.083991 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.083964 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-kube-api-access-r2lkh" pod "e3ea931d-3fd7-4c23-994b-902b8cedfe2a" (UID: "e3ea931d-3fd7-4c23-994b-902b8cedfe2a"). InnerVolumeSpecName "kube-api-access-r2lkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:06.086806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.086781 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-xks-ca" pod "e3ea931d-3fd7-4c23-994b-902b8cedfe2a" (UID: "e3ea931d-3fd7-4c23-994b-902b8cedfe2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:06.182624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.182594 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:06.182624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.182620 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lkh\" (UniqueName: \"kubernetes.io/projected/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-kube-api-access-r2lkh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:06.182624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.182629 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ea931d-3fd7-4c23-994b-902b8cedfe2a-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:06.276549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.268846 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:54:06.276549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.268903 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:54:06.776886 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.776855 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" Oct 30 00:54:06.777064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.776888 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk" event={"ID":"e3ea931d-3fd7-4c23-994b-902b8cedfe2a","Type":"ContainerDied","Data":"125669ba23ba7cd7931aa01cd19d4ae3fa8841dd461b6feaf8751a2099eff942"} Oct 30 00:54:06.777064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:06.776941 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="125669ba23ba7cd7931aa01cd19d4ae3fa8841dd461b6feaf8751a2099eff942" Oct 30 00:54:07.265105 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:07.265072 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:54:07.265571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:07.265136 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:54:08.449547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:08.444368 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:08.449547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:08.445948 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:08.458541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:08.452807 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 00:54:09.583545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.582914 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7"] Oct 30 00:54:09.584057 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.583779 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e3ea931d-3fd7-4c23-994b-902b8cedfe2a" containerName="revoke-keys" Oct 30 00:54:09.584057 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.583800 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ea931d-3fd7-4c23-994b-902b8cedfe2a" containerName="revoke-keys" Oct 30 00:54:09.584057 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.583935 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ea931d-3fd7-4c23-994b-902b8cedfe2a" containerName="revoke-keys" Oct 30 00:54:09.713748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.713707 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7"] Oct 30 00:54:09.746542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.722695 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.846843 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tls-certs\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.302729 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx674\" (UniqueName: \"kubernetes.io/projected/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-kube-api-access-xx674\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.302773 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-home\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.302846 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-dshm\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.302964 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-model-cache\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.303067 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tmp-dir\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.846929 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.242831 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.300107 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.029773 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:54:10.321722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:09.999385 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:10.386018 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.385985 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"distinct-w6-5a344d-kserve-self-signed-certs\"" Oct 30 00:54:10.417539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.409569 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tmp-dir\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.417539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.409677 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tls-certs\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.417539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.409738 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx674\" (UniqueName: \"kubernetes.io/projected/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-kube-api-access-xx674\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.417539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.409928 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:10.417539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.411262 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tmp-dir\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.422538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.417971 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-home\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.422538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.418057 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-dshm\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.422538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.418151 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-model-cache\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.422538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.418580 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-model-cache\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.422538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.419091 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-home\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.557235 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.557206 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tls-certs\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.619633 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.600488 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx674\" (UniqueName: \"kubernetes.io/projected/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-kube-api-access-xx674\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.621256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.621209 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-dshm\") pod \"distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:10.921590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:10.920762 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:11.312679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.312630 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:11.313571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.313549 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:54:11.316163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.316137 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:54:11.316635 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.316617 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:54:11.317688 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.317601 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:11.318215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.318183 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:54:11.322079 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.322055 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:54:11.322564 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.322244 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:11.323371 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.322725 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:54:11.334193 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.334169 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:54:11.596486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.596448 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 30 00:54:11.596940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.596913 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:54:11.600544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.597379 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:54:11.986606 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.984879 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:54:11.986606 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.986006 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:54:11.986606 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:11.986060 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:54:12.007201 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.003991 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:54:12.010562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.008952 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:54:12.022286 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.020958 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:54:12.493542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.492701 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7"] Oct 30 00:54:12.815774 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.814718 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" event={"ID":"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626","Type":"ContainerStarted","Data":"4f5eea68d99dc29c20e900607246c6f66648eb3b184756517f977451be6ed9d5"} Oct 30 00:54:12.910619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.910577 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz"] Oct 30 00:54:12.936550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.930611 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz"] Oct 30 00:54:12.936550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.930780 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:12.950773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.950081 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:54:12.950773 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.950326 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:54:12.954539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.951170 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:54:12.954539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:12.951322 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:54:12.996935 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:54:12.996897 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:54:13.096676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.096347 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.096676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.096447 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95rw\" (UniqueName: \"kubernetes.io/projected/a8262f00-9ea2-4828-acb7-707c32b9f355-kube-api-access-z95rw\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.096676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.096563 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.199919 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.198424 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.199919 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.198508 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95rw\" (UniqueName: \"kubernetes.io/projected/a8262f00-9ea2-4828-acb7-707c32b9f355-kube-api-access-z95rw\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.199919 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.198602 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.199919 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.199497 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.203874 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.203848 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.223390 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.223360 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95rw\" (UniqueName: \"kubernetes.io/projected/a8262f00-9ea2-4828-acb7-707c32b9f355-kube-api-access-z95rw\") pod \"maas-api-revoke-sub-3e1d4f71-qjtzz\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.266548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.265838 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:13.544562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.522724 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh"] Oct 30 00:54:13.625753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.619247 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh"] Oct 30 00:54:13.625753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.619369 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.713479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.713404 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.713479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.713468 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.713687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.713629 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbt8\" (UniqueName: \"kubernetes.io/projected/707e1469-699c-4fc5-8975-53d4e5392334-kube-api-access-msbt8\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.815249 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.815194 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.815407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.815339 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbt8\" (UniqueName: \"kubernetes.io/projected/707e1469-699c-4fc5-8975-53d4e5392334-kube-api-access-msbt8\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.815501 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.815452 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.816049 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.816025 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.817776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.817751 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.822054 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.822026 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" event={"ID":"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626","Type":"ContainerStarted","Data":"70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9"} Oct 30 00:54:13.837783 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.837761 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbt8\" (UniqueName: \"kubernetes.io/projected/707e1469-699c-4fc5-8975-53d4e5392334-kube-api-access-msbt8\") pod \"maas-api-revoke-sub-d4ad2fd1-7bvhh\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:13.851233 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:54:13.851209 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8262f00_9ea2_4828_acb7_707c32b9f355.slice/crio-84ec01d6533efaec5c79b7bac4d918999ae6aa498dcf12e384c191ca286735ef WatchSource:0}: Error finding container 84ec01d6533efaec5c79b7bac4d918999ae6aa498dcf12e384c191ca286735ef: Status 404 returned error can't find the container with id 84ec01d6533efaec5c79b7bac4d918999ae6aa498dcf12e384c191ca286735ef Oct 30 00:54:13.852236 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.852217 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz"] Oct 30 00:54:13.910480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.909754 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podStartSLOduration=4.909735015 podStartE2EDuration="4.909735015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:13.905393814 +0000 UTC m=+938.848567006" watchObservedRunningTime="2026-10-30 00:54:13.909735015 +0000 UTC m=+938.852908208" Oct 30 00:54:13.928479 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.927710 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:13.946168 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:13.946141 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:14.327988 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:54:14.327963 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707e1469_699c_4fc5_8975_53d4e5392334.slice/crio-710e1c00a0d1232959ca54281a84525ae90b1062860bfb6b482993cdc59d2463 WatchSource:0}: Error finding container 710e1c00a0d1232959ca54281a84525ae90b1062860bfb6b482993cdc59d2463: Status 404 returned error can't find the container with id 710e1c00a0d1232959ca54281a84525ae90b1062860bfb6b482993cdc59d2463 Oct 30 00:54:14.329489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:14.329465 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh"] Oct 30 00:54:14.831856 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:14.831769 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" event={"ID":"a8262f00-9ea2-4828-acb7-707c32b9f355","Type":"ContainerStarted","Data":"ac1b411b622a21fae6623f7dd241348b523b4092bde1d2c9492a44170bb30df7"} Oct 30 00:54:14.831856 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:14.831824 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" event={"ID":"a8262f00-9ea2-4828-acb7-707c32b9f355","Type":"ContainerStarted","Data":"84ec01d6533efaec5c79b7bac4d918999ae6aa498dcf12e384c191ca286735ef"} Oct 30 00:54:14.833386 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:14.833360 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" event={"ID":"707e1469-699c-4fc5-8975-53d4e5392334","Type":"ContainerStarted","Data":"1df0d2863f1e22d4583edacfe9270ff71bc7bd37bc8d2041b8fd357a3e974729"} Oct 30 00:54:14.833504 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:14.833408 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" event={"ID":"707e1469-699c-4fc5-8975-53d4e5392334","Type":"ContainerStarted","Data":"710e1c00a0d1232959ca54281a84525ae90b1062860bfb6b482993cdc59d2463"} Oct 30 00:54:14.872104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:14.872054 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" podStartSLOduration=2.872041717 podStartE2EDuration="2.872041717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:14.871089021 +0000 UTC m=+939.814262213" watchObservedRunningTime="2026-10-30 00:54:14.872041717 +0000 UTC m=+939.815214908" Oct 30 00:54:14.912619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:14.912575 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" podStartSLOduration=1.912560893 podStartE2EDuration="1.912560893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:14.910923753 +0000 UTC m=+939.854096945" watchObservedRunningTime="2026-10-30 00:54:14.912560893 +0000 UTC m=+939.855734087" Oct 30 00:54:15.841244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:15.841206 4905 generic.go:356] "Generic (PLEG): container finished" podID="a8262f00-9ea2-4828-acb7-707c32b9f355" containerID="ac1b411b622a21fae6623f7dd241348b523b4092bde1d2c9492a44170bb30df7" exitCode=0 Oct 30 00:54:15.841701 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:15.841291 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" event={"ID":"a8262f00-9ea2-4828-acb7-707c32b9f355","Type":"ContainerDied","Data":"ac1b411b622a21fae6623f7dd241348b523b4092bde1d2c9492a44170bb30df7"} Oct 30 00:54:15.842847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:15.842826 4905 generic.go:356] "Generic (PLEG): container finished" podID="707e1469-699c-4fc5-8975-53d4e5392334" containerID="1df0d2863f1e22d4583edacfe9270ff71bc7bd37bc8d2041b8fd357a3e974729" exitCode=0 Oct 30 00:54:15.842964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:15.842898 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" event={"ID":"707e1469-699c-4fc5-8975-53d4e5392334","Type":"ContainerDied","Data":"1df0d2863f1e22d4583edacfe9270ff71bc7bd37bc8d2041b8fd357a3e974729"} Oct 30 00:54:17.069384 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.069356 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:17.072987 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.072966 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:17.161019 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.160948 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-openshift-ca\") pod \"707e1469-699c-4fc5-8975-53d4e5392334\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " Oct 30 00:54:17.161167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.161035 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-xks-ca\") pod \"707e1469-699c-4fc5-8975-53d4e5392334\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " Oct 30 00:54:17.161167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.161054 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707e1469-699c-4fc5-8975-53d4e5392334-kube-api-access-msbt8\" (UniqueName: \"kubernetes.io/projected/707e1469-699c-4fc5-8975-53d4e5392334-kube-api-access-msbt8\") pod \"707e1469-699c-4fc5-8975-53d4e5392334\" (UID: \"707e1469-699c-4fc5-8975-53d4e5392334\") " Oct 30 00:54:17.161410 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.161377 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-openshift-ca" pod "707e1469-699c-4fc5-8975-53d4e5392334" (UID: "707e1469-699c-4fc5-8975-53d4e5392334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:17.161559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.161542 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:17.163340 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.163316 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-xks-ca" pod "707e1469-699c-4fc5-8975-53d4e5392334" (UID: "707e1469-699c-4fc5-8975-53d4e5392334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:17.163801 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.163776 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707e1469-699c-4fc5-8975-53d4e5392334-kube-api-access-msbt8" pod "707e1469-699c-4fc5-8975-53d4e5392334" (UID: "707e1469-699c-4fc5-8975-53d4e5392334"). InnerVolumeSpecName "kube-api-access-msbt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:17.262764 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.262732 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8262f00-9ea2-4828-acb7-707c32b9f355-kube-api-access-z95rw\" (UniqueName: \"kubernetes.io/projected/a8262f00-9ea2-4828-acb7-707c32b9f355-kube-api-access-z95rw\") pod \"a8262f00-9ea2-4828-acb7-707c32b9f355\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " Oct 30 00:54:17.262764 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.262772 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-openshift-ca\") pod \"a8262f00-9ea2-4828-acb7-707c32b9f355\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " Oct 30 00:54:17.263056 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.262868 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-xks-ca\") pod \"a8262f00-9ea2-4828-acb7-707c32b9f355\" (UID: \"a8262f00-9ea2-4828-acb7-707c32b9f355\") " Oct 30 00:54:17.263167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.263142 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-openshift-ca" pod "a8262f00-9ea2-4828-acb7-707c32b9f355" (UID: "a8262f00-9ea2-4828-acb7-707c32b9f355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:17.263334 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.263320 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707e1469-699c-4fc5-8975-53d4e5392334-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:17.263387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.263338 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbt8\" (UniqueName: \"kubernetes.io/projected/707e1469-699c-4fc5-8975-53d4e5392334-kube-api-access-msbt8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:17.263387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.263347 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:17.265046 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.265024 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8262f00-9ea2-4828-acb7-707c32b9f355-kube-api-access-z95rw" pod "a8262f00-9ea2-4828-acb7-707c32b9f355" (UID: "a8262f00-9ea2-4828-acb7-707c32b9f355"). InnerVolumeSpecName "kube-api-access-z95rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:17.265178 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.265144 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-xks-ca" pod "a8262f00-9ea2-4828-acb7-707c32b9f355" (UID: "a8262f00-9ea2-4828-acb7-707c32b9f355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:17.364497 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.364461 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8262f00-9ea2-4828-acb7-707c32b9f355-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:17.364497 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.364490 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95rw\" (UniqueName: \"kubernetes.io/projected/a8262f00-9ea2-4828-acb7-707c32b9f355-kube-api-access-z95rw\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:54:17.856614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.856584 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" Oct 30 00:54:17.856779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.856610 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz" event={"ID":"a8262f00-9ea2-4828-acb7-707c32b9f355","Type":"ContainerDied","Data":"84ec01d6533efaec5c79b7bac4d918999ae6aa498dcf12e384c191ca286735ef"} Oct 30 00:54:17.856779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.856658 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ec01d6533efaec5c79b7bac4d918999ae6aa498dcf12e384c191ca286735ef" Oct 30 00:54:17.858071 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.858044 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" event={"ID":"707e1469-699c-4fc5-8975-53d4e5392334","Type":"ContainerDied","Data":"710e1c00a0d1232959ca54281a84525ae90b1062860bfb6b482993cdc59d2463"} Oct 30 00:54:17.858188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.858078 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh" Oct 30 00:54:17.858188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:17.858093 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710e1c00a0d1232959ca54281a84525ae90b1062860bfb6b482993cdc59d2463" Oct 30 00:54:20.922107 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:20.922053 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:20.935134 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:20.935105 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:21.916754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:21.916723 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:54:24.956545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:24.955856 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:54:25.061986 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:25.061606 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:30.325007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.324971 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6"] Oct 30 00:54:30.330927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.330895 4905 cpu_manager.go:414] "containerMap: removing container" podUID="707e1469-699c-4fc5-8975-53d4e5392334" containerName="revoke-keys" Oct 30 00:54:30.330927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.330922 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707e1469-699c-4fc5-8975-53d4e5392334" containerName="revoke-keys" Oct 30 00:54:30.331119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.330962 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a8262f00-9ea2-4828-acb7-707c32b9f355" containerName="revoke-keys" Oct 30 00:54:30.331119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.330969 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8262f00-9ea2-4828-acb7-707c32b9f355" containerName="revoke-keys" Oct 30 00:54:30.331119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.331089 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="707e1469-699c-4fc5-8975-53d4e5392334" containerName="revoke-keys" Oct 30 00:54:30.331119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.331105 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8262f00-9ea2-4828-acb7-707c32b9f355" containerName="revoke-keys" Oct 30 00:54:30.335466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.335442 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.362825 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.362797 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6"] Oct 30 00:54:30.384954 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.384922 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385122 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.384980 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.385144 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.385279 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385414 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.385397 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.385454 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43af3985-3f75-48bb-92a6-b025bdc61f8a-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.385555 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.385597 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdn74\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-kube-api-access-qdn74\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.385748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.385717 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486443 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486405 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486618 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486499 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486618 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486587 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43af3985-3f75-48bb-92a6-b025bdc61f8a-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486616 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486644 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdn74\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-kube-api-access-qdn74\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486721 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486770 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.486966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486828 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.487078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486971 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.487078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.486997 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.487255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.487232 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.487366 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.487340 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.487500 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.487403 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.487500 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.487429 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43af3985-3f75-48bb-92a6-b025bdc61f8a-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.489245 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.489216 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.489363 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.489339 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.508481 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.508369 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.508614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.508497 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdn74\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-kube-api-access-qdn74\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.652600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.652506 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:30.855486 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.855444 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2"] Oct 30 00:54:30.861228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.861185 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.882705 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.882678 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2"] Oct 30 00:54:30.891477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.891447 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.891613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.891533 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.891692 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.891668 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.891813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.891789 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.891888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.891823 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.891888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.891849 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9nnh\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-kube-api-access-m9nnh\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.892026 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.891997 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.892084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.892053 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.892143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.892088 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.972481 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:54:30.972453 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43af3985_3f75_48bb_92a6_b025bdc61f8a.slice/crio-69826887693fad294a83768bd321883c48fa127ba3377121fdc7d4ff5f3c6e94 WatchSource:0}: Error finding container 69826887693fad294a83768bd321883c48fa127ba3377121fdc7d4ff5f3c6e94: Status 404 returned error can't find the container with id 69826887693fad294a83768bd321883c48fa127ba3377121fdc7d4ff5f3c6e94 Oct 30 00:54:30.973798 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.973764 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6"] Oct 30 00:54:30.975724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.975690 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:30.975840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.975767 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:30.975840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.975800 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:30.993606 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.993563 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.993712 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.993652 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.993775 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.993713 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.993831 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.993778 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994098 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994072 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994166 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994155 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994490 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994469 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994634 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994681 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994709 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9nnh\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-kube-api-access-m9nnh\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994839 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994823 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.994898 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.994867 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.995061 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.995022 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.995258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.995219 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.996187 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.996160 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:30.997191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:30.997170 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:31.015391 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.015351 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:31.015875 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.015854 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9nnh\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-kube-api-access-m9nnh\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:31.027048 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.027022 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" event={"ID":"43af3985-3f75-48bb-92a6-b025bdc61f8a","Type":"ContainerStarted","Data":"69826887693fad294a83768bd321883c48fa127ba3377121fdc7d4ff5f3c6e94"} Oct 30 00:54:31.182279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.182247 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:31.604852 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:54:31.604818 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a0ddfc_9234_49cf_ae53_41b9cd22ee7b.slice/crio-f2519c38f9d1495d3626e11b87b7f5ca336701e2a1697fb37df0c7cd1f575001 WatchSource:0}: Error finding container f2519c38f9d1495d3626e11b87b7f5ca336701e2a1697fb37df0c7cd1f575001: Status 404 returned error can't find the container with id f2519c38f9d1495d3626e11b87b7f5ca336701e2a1697fb37df0c7cd1f575001 Oct 30 00:54:31.605578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.605549 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2"] Oct 30 00:54:31.607357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.607324 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:31.607440 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.607384 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:31.607440 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.607413 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:31.864598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.864426 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm"] Oct 30 00:54:31.870687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.870661 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.891387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.891342 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm"] Oct 30 00:54:31.917841 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.917750 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.917997 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.917870 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575fa231-7d17-4522-bd3e-cc8bebe2e964-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.917997 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.917911 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.917997 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.917955 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlh4\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-kube-api-access-2zlh4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.918162 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.918055 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.918293 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.918274 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.918393 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.918376 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.918470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.918454 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:31.918537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:31.918484 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019371 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019334 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019405 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019447 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019581 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019607 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575fa231-7d17-4522-bd3e-cc8bebe2e964-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019642 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019668 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlh4\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-kube-api-access-2zlh4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019705 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019977 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019822 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.019977 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.019917 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.020387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.020356 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.020497 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.020471 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575fa231-7d17-4522-bd3e-cc8bebe2e964-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.020673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.020648 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.020776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.020734 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.023191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.023164 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.068543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.050082 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.073591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.073482 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.075291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.075260 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlh4\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-kube-api-access-2zlh4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-g5ltm\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.079656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.079066 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" event={"ID":"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b","Type":"ContainerStarted","Data":"49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9"} Oct 30 00:54:32.079656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.079123 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" event={"ID":"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b","Type":"ContainerStarted","Data":"f2519c38f9d1495d3626e11b87b7f5ca336701e2a1697fb37df0c7cd1f575001"} Oct 30 00:54:32.084473 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.084453 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" event={"ID":"43af3985-3f75-48bb-92a6-b025bdc61f8a","Type":"ContainerStarted","Data":"93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe"} Oct 30 00:54:32.147606 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.145102 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podStartSLOduration=2.145081222 podStartE2EDuration="2.145081222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:32.122171904 +0000 UTC m=+957.065345105" watchObservedRunningTime="2026-10-30 00:54:32.145081222 +0000 UTC m=+957.088254417" Oct 30 00:54:32.171894 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.171844 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podStartSLOduration=2.171830764 podStartE2EDuration="2.171830764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:32.171043692 +0000 UTC m=+957.114216907" watchObservedRunningTime="2026-10-30 00:54:32.171830764 +0000 UTC m=+957.115003954" Oct 30 00:54:32.183130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.183106 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:32.184611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.184588 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:32.184728 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.184630 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:32.191146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.191118 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:32.198771 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.198745 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2"] Oct 30 00:54:32.203778 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.203756 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.223020 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.220955 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"distinct-w4-b565ec-kserve-self-signed-certs\"" Oct 30 00:54:32.247551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.246930 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2"] Oct 30 00:54:32.345369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.344686 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab69f74-0877-4f68-a0dd-b4a631fead70-tls-certs\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.345369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.344747 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-model-cache\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.345369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.344899 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-tmp-dir\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.345369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.344958 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-home\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.345369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.344988 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bh2z\" (UniqueName: \"kubernetes.io/projected/3ab69f74-0877-4f68-a0dd-b4a631fead70-kube-api-access-4bh2z\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.345369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.345148 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-dshm\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.508545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.499624 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-dshm\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.508545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.499849 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab69f74-0877-4f68-a0dd-b4a631fead70-tls-certs\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.508545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.499933 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-model-cache\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.508545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.500154 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-tmp-dir\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.508545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.500221 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-home\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.508545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.500248 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bh2z\" (UniqueName: \"kubernetes.io/projected/3ab69f74-0877-4f68-a0dd-b4a631fead70-kube-api-access-4bh2z\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.508545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.502115 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-model-cache\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.510483 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.510450 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-tmp-dir\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.510775 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.510753 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-home\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.544077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.544048 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab69f74-0877-4f68-a0dd-b4a631fead70-tls-certs\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.544672 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.544629 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-dshm\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.586337 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.586308 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bh2z\" (UniqueName: \"kubernetes.io/projected/3ab69f74-0877-4f68-a0dd-b4a631fead70-kube-api-access-4bh2z\") pod \"distinct-w4-b565ec-kserve-6bc489b89-sj4l2\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:32.609966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.609928 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm"] Oct 30 00:54:32.631586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.631511 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:32.631882 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.631857 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:32.632030 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.632011 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:32.678577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.654286 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:32.678577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.654412 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:54:32.678577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.654454 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:54:32.846275 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:32.846004 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:33.130567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.124202 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" event={"ID":"575fa231-7d17-4522-bd3e-cc8bebe2e964","Type":"ContainerStarted","Data":"f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26"} Oct 30 00:54:33.130567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.124260 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" event={"ID":"575fa231-7d17-4522-bd3e-cc8bebe2e964","Type":"ContainerStarted","Data":"fc92bb083164210303ccf6992ce517a01e4aa6721d9f4e941f0cf69dffc9fe39"} Oct 30 00:54:33.204545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.191104 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:33.204545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.191160 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:33.222374 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.222311 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podStartSLOduration=2.2222918050000002 podStartE2EDuration="2.222291805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:33.207634635 +0000 UTC m=+958.150807827" watchObservedRunningTime="2026-10-30 00:54:33.222291805 +0000 UTC m=+958.165464997" Oct 30 00:54:33.234553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.222648 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:33.264545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.261690 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:33.264545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.261743 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:33.691745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.688613 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:54:33.691745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.688664 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:54:33.782245 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:33.769423 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2"] Oct 30 00:54:34.200137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:34.200033 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:34.228698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:34.228656 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:34.242862 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:34.200639 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:34.243087 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:34.243058 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:34.302546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:34.298508 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" event={"ID":"3ab69f74-0877-4f68-a0dd-b4a631fead70","Type":"ContainerStarted","Data":"e4ea840dbaad991eabf1d87eac03cbe573dd1b9bf5f91d2e849413a208951f18"} Oct 30 00:54:34.653076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:34.653040 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:54:34.654388 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:34.654341 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:54:35.190543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.188546 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:35.190543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.188601 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:35.198559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.196618 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:35.198559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.196672 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:35.364560 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.364472 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" event={"ID":"3ab69f74-0877-4f68-a0dd-b4a631fead70","Type":"ContainerStarted","Data":"a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96"} Oct 30 00:54:35.653541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.653179 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:54:35.653541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.653240 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:54:35.848543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:35.846692 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:36.207545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:36.183948 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:36.207545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:36.184457 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:36.207545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:36.191796 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:36.207545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:36.191837 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:36.656545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:36.655885 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:54:36.656545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:36.655947 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:54:37.207552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:37.192478 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:37.207552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:37.192557 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:37.207552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:37.193140 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:37.207552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:37.193180 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:37.658546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:37.655971 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:54:37.658546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:37.656037 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:54:38.183909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:38.183873 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:38.184217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:38.184192 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:38.191530 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:38.191494 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:38.191681 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:38.191664 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:38.908162 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:38.908136 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:38.909318 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:38.909299 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:38.909864 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:38.909845 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 00:54:39.005251 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:39.005202 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podStartSLOduration=7.005189525 podStartE2EDuration="7.005189525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:35.457355994 +0000 UTC m=+960.400529186" watchObservedRunningTime="2026-10-30 00:54:39.005189525 +0000 UTC m=+963.948362715" Oct 30 00:54:39.188547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:39.188438 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:54:39.188813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:39.188769 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:54:39.191422 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:39.191396 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:39.193397 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:39.193276 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:40.191577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:40.191513 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:40.191936 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:40.191603 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:40.538147 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:40.538115 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:40.539427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:40.539401 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:40.540965 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:40.540945 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 00:54:41.192627 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:41.192588 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:54:41.193139 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:41.192658 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:54:42.415619 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:54:42.415581 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:54:42.443174 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:42.443137 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:42.443739 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:42.443713 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:42.446131 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:42.446114 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 00:54:42.846896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:42.846864 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:42.860311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:42.860285 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:43.566458 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:43.566424 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:54:45.543942 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.543909 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8"] Oct 30 00:54:45.549572 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.549546 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.602505 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.602471 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8"] Oct 30 00:54:45.710466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710435 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710659 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710474 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710659 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710504 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520fcf61-f470-433f-8767-58d971b4e9a9-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710746 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710691 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710746 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710720 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710750 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctzt\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-kube-api-access-gctzt\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710816 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710915 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710842 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.710953 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.710920 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520fcf61-f470-433f-8767-58d971b4e9a9-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.811854 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.811779 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812010 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.811861 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520fcf61-f470-433f-8767-58d971b4e9a9-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812010 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.811972 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812010 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812006 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812187 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812047 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520fcf61-f470-433f-8767-58d971b4e9a9-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812187 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812134 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812187 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812163 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812347 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812193 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctzt\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-kube-api-access-gctzt\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812347 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812232 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812615 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812743 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812642 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520fcf61-f470-433f-8767-58d971b4e9a9-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812743 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812659 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.812840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812820 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.813024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.812958 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.814302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.814269 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.814594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.814577 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520fcf61-f470-433f-8767-58d971b4e9a9-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.832218 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.832197 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.832722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.832700 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctzt\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-kube-api-access-gctzt\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:45.871015 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:45.870993 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:46.180874 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:54:46.180845 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520fcf61_f470_433f_8767_58d971b4e9a9.slice/crio-cf06b3b37546ebf9d2cdd15e5b04792f9eaf73c511ff604e7ed1d09178d2c504 WatchSource:0}: Error finding container cf06b3b37546ebf9d2cdd15e5b04792f9eaf73c511ff604e7ed1d09178d2c504: Status 404 returned error can't find the container with id cf06b3b37546ebf9d2cdd15e5b04792f9eaf73c511ff604e7ed1d09178d2c504 Oct 30 00:54:46.182734 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.182706 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8"] Oct 30 00:54:46.183324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.183291 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:46.183425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.183356 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:46.183425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.183384 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:54:46.595388 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.595356 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" event={"ID":"520fcf61-f470-433f-8767-58d971b4e9a9","Type":"ContainerStarted","Data":"3ec30f8bc037bcc95e6b06cdf6d5bd0fd182c07ac744b4402f9bd54a1c069ebf"} Oct 30 00:54:46.595938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.595409 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" event={"ID":"520fcf61-f470-433f-8767-58d971b4e9a9","Type":"ContainerStarted","Data":"cf06b3b37546ebf9d2cdd15e5b04792f9eaf73c511ff604e7ed1d09178d2c504"} Oct 30 00:54:46.640569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.640503 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podStartSLOduration=1.640489111 podStartE2EDuration="1.640489111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:46.638471266 +0000 UTC m=+971.581644458" watchObservedRunningTime="2026-10-30 00:54:46.640489111 +0000 UTC m=+971.583662302" Oct 30 00:54:46.871737 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:46.871647 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:47.885545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:47.883368 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:54:47.885545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:47.883442 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:54:48.872052 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:48.872012 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:54:48.872219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:48.872064 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:54:49.871621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:49.871587 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:54:49.872024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:49.871656 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:54:50.871613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:50.871578 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:54:50.871813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:50.871637 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:54:51.871928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:51.871892 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:54:51.872439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:51.871944 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:54:53.178388 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:53.178349 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:53.179129 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:53.179105 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:53.179982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:53.179959 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 00:54:54.389932 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.389894 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt"] Oct 30 00:54:54.395067 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.395046 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.412679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.412656 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt"] Oct 30 00:54:54.422212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.422189 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"distinct-2-w0-3c2596-kserve-self-signed-certs\"" Oct 30 00:54:54.519956 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.519923 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-home\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.520130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.519960 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-model-cache\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.520130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.520089 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-tmp-dir\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.520264 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.520237 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-dshm\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.520325 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.520306 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f19f52-412e-4c09-8323-6fed84a9a68b-tls-certs\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.520377 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.520358 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq72g\" (UniqueName: \"kubernetes.io/projected/c6f19f52-412e-4c09-8323-6fed84a9a68b-kube-api-access-fq72g\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.621424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.621381 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f19f52-412e-4c09-8323-6fed84a9a68b-tls-certs\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.622251 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.622222 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq72g\" (UniqueName: \"kubernetes.io/projected/c6f19f52-412e-4c09-8323-6fed84a9a68b-kube-api-access-fq72g\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.622893 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.622871 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-home\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.623408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.623390 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-model-cache\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.636670 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.623343 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-home\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.636913 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.632725 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-model-cache\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.636913 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.636712 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-tmp-dir\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.637109 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.637012 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-tmp-dir\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.637109 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.637017 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-dshm\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.638793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.638770 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f19f52-412e-4c09-8323-6fed84a9a68b-tls-certs\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.651127 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.651059 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-dshm\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.671314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.671235 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq72g\" (UniqueName: \"kubernetes.io/projected/c6f19f52-412e-4c09-8323-6fed84a9a68b-kube-api-access-fq72g\") pod \"distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:54.714485 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:54.714422 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:54:55.033230 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:54:55.033199 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6f19f52_412e_4c09_8323_6fed84a9a68b.slice/crio-d207c2fee09d60039d1f10288509991c0936233c09eafbbfa225d189b943a849 WatchSource:0}: Error finding container d207c2fee09d60039d1f10288509991c0936233c09eafbbfa225d189b943a849: Status 404 returned error can't find the container with id d207c2fee09d60039d1f10288509991c0936233c09eafbbfa225d189b943a849 Oct 30 00:54:55.034995 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:55.034906 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt"] Oct 30 00:54:55.719387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:55.719350 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" event={"ID":"c6f19f52-412e-4c09-8323-6fed84a9a68b","Type":"ContainerStarted","Data":"7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3"} Oct 30 00:54:55.719870 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:55.719401 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" event={"ID":"c6f19f52-412e-4c09-8323-6fed84a9a68b","Type":"ContainerStarted","Data":"d207c2fee09d60039d1f10288509991c0936233c09eafbbfa225d189b943a849"} Oct 30 00:54:55.766553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:55.766072 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podStartSLOduration=1.7660537330000001 podStartE2EDuration="1.766053733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:55.762927763 +0000 UTC m=+980.706100958" watchObservedRunningTime="2026-10-30 00:54:55.766053733 +0000 UTC m=+980.709226926" Oct 30 00:54:56.715547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:54:56.715503 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:55:04.715477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:04.715446 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:55:04.728621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:04.728595 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:55:04.793044 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:04.793012 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:55:12.503783 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:55:12.503739 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:55:14.197709 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:14.197662 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:14.799664 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:14.799605 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:14.900646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:14.900596 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:55:14.936814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:14.936776 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:55:14.965891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:14.965853 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:14.986916 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:14.985834 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:55:15.031646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:15.031602 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:15.199865 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:15.199769 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:15.212924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:15.212893 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:55:15.517996 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:15.517962 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:55:15.557651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:15.557610 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:15.717034 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:15.716996 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:19.638829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.638790 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt"] Oct 30 00:55:19.645605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.645580 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.666175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.666150 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt"] Oct 30 00:55:19.675263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.675243 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"default-dockercfg-jtgwd\"" Oct 30 00:55:19.675394 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.675262 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"distinct-2-w5-da76a2-kserve-self-signed-certs\"" Oct 30 00:55:19.676017 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.676002 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"openshift-service-ca.crt\"" Oct 30 00:55:19.676359 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.676326 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"kube-root-ca.crt\"" Oct 30 00:55:19.749730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.749703 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tls-certs\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.749851 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.749770 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-home\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.749851 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.749810 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-model-cache\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.749924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.749902 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-dshm\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.750053 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.750039 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tmp-dir\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.750102 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.750090 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh97r\" (UniqueName: \"kubernetes.io/projected/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-kube-api-access-bh97r\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.853405 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.853365 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tmp-dir\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.879726 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.879687 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh97r\" (UniqueName: \"kubernetes.io/projected/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-kube-api-access-bh97r\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.879990 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.879971 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tls-certs\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.880171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.880159 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-home\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.880255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.880245 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-model-cache\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.880363 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.880353 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-dshm\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.880981 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.880958 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-model-cache\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.881109 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.881055 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-home\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.881188 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.856958 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tmp-dir\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.891992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.891921 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-dshm\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.895292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.895217 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tls-certs\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.922539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.922489 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh97r\" (UniqueName: \"kubernetes.io/projected/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-kube-api-access-bh97r\") pod \"distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:19.961051 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:19.960827 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:20.292971 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:20.292939 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e29d6bb_e022_4b06_ae02_5b964fc5d34e.slice/crio-945a08f8d4abf3943499c19bd342202bd4f2f78ed7c6f89a1553775f32979e4b WatchSource:0}: Error finding container 945a08f8d4abf3943499c19bd342202bd4f2f78ed7c6f89a1553775f32979e4b: Status 404 returned error can't find the container with id 945a08f8d4abf3943499c19bd342202bd4f2f78ed7c6f89a1553775f32979e4b Oct 30 00:55:20.293956 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:20.293932 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt"] Oct 30 00:55:21.058822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:21.057186 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" event={"ID":"5e29d6bb-e022-4b06-ae02-5b964fc5d34e","Type":"ContainerStarted","Data":"bf104aa77587f493e2a7ad1e56b408ce458e8fed3da2d787c7ffea1dce1783d8"} Oct 30 00:55:21.058822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:21.057245 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" event={"ID":"5e29d6bb-e022-4b06-ae02-5b964fc5d34e","Type":"ContainerStarted","Data":"945a08f8d4abf3943499c19bd342202bd4f2f78ed7c6f89a1553775f32979e4b"} Oct 30 00:55:21.112184 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:21.112130 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podStartSLOduration=2.112114584 podStartE2EDuration="2.112114584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:21.110760356 +0000 UTC m=+1006.053933550" watchObservedRunningTime="2026-10-30 00:55:21.112114584 +0000 UTC m=+1006.055287781" Oct 30 00:55:21.961281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:21.961246 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:29.961769 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:29.961734 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:29.977774 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:29.977748 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:30.168341 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.168308 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:55:30.666754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.666717 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr"] Oct 30 00:55:30.672717 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.672693 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.700874 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.700843 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr"] Oct 30 00:55:30.846177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.846144 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6"] Oct 30 00:55:30.850828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.850791 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.859921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.859892 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.859940 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08147324-90f2-4d15-902c-c2e0d45eed52-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.860386 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860481 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.860450 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.860509 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860641 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.860625 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08147324-90f2-4d15-902c-c2e0d45eed52-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860683 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.860669 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860758 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.860743 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt26s\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-kube-api-access-kt26s\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.860831 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.860816 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.874654 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.874632 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6"] Oct 30 00:55:30.961955 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.961865 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.961955 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.961912 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.961955 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.961947 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.961994 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcl9\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-kube-api-access-zhcl9\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962124 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962162 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962193 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08147324-90f2-4d15-902c-c2e0d45eed52-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962322 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962345 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948f189d-119b-4657-ab78-81373fffd7e3-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962389 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948f189d-119b-4657-ab78-81373fffd7e3-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962426 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962457 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962494 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.962563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962566 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.963142 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962583 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.963142 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962593 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:30.963142 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.962916 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.963301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.963166 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.963357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.963325 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08147324-90f2-4d15-902c-c2e0d45eed52-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.963417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.963363 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.963417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.963395 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt26s\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-kube-api-access-kt26s\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.964382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.964355 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08147324-90f2-4d15-902c-c2e0d45eed52-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.964609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.964590 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.966752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.966732 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08147324-90f2-4d15-902c-c2e0d45eed52-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.966843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.966824 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.982683 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.982650 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.983395 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.983376 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt26s\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-kube-api-access-kt26s\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:30.993837 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:30.993817 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.080092 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948f189d-119b-4657-ab78-81373fffd7e3-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.090549 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948f189d-119b-4657-ab78-81373fffd7e3-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.090621 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948f189d-119b-4657-ab78-81373fffd7e3-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.094276 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948f189d-119b-4657-ab78-81373fffd7e3-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.094830 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.094897 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.094926 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.095062 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.095096 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.095164 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcl9\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-kube-api-access-zhcl9\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.095576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.095240 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.096215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.095757 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.096582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.096558 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.096886 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.096860 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.097917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.097894 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.100472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.100316 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.306003 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.292711 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.376915 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:31.376883 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08147324_90f2_4d15_902c_c2e0d45eed52.slice/crio-f22b7317c18571927f86c443e4722ffc1aabb474539aa66d16e7bb4f43e0d74d WatchSource:0}: Error finding container f22b7317c18571927f86c443e4722ffc1aabb474539aa66d16e7bb4f43e0d74d: Status 404 returned error can't find the container with id f22b7317c18571927f86c443e4722ffc1aabb474539aa66d16e7bb4f43e0d74d Oct 30 00:55:31.379172 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.379138 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcl9\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-kube-api-access-zhcl9\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.379613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.379582 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:31.379725 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.379669 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:31.379824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.379719 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:31.381006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.380980 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr"] Oct 30 00:55:31.466637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.466604 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:31.887426 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:31.886338 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948f189d_119b_4657_ab78_81373fffd7e3.slice/crio-af8e93789f976e313fc0bdd6d638625ec1a52a8a62740f0f9c09e32cd2b0178a WatchSource:0}: Error finding container af8e93789f976e313fc0bdd6d638625ec1a52a8a62740f0f9c09e32cd2b0178a: Status 404 returned error can't find the container with id af8e93789f976e313fc0bdd6d638625ec1a52a8a62740f0f9c09e32cd2b0178a Oct 30 00:55:31.888343 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.888320 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6"] Oct 30 00:55:31.889737 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.889693 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:31.889847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.889781 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:31.889847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:31.889827 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:32.431425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.431387 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" event={"ID":"948f189d-119b-4657-ab78-81373fffd7e3","Type":"ContainerStarted","Data":"3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b"} Oct 30 00:55:32.432064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.432037 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" event={"ID":"948f189d-119b-4657-ab78-81373fffd7e3","Type":"ContainerStarted","Data":"af8e93789f976e313fc0bdd6d638625ec1a52a8a62740f0f9c09e32cd2b0178a"} Oct 30 00:55:32.447917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.447883 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" event={"ID":"08147324-90f2-4d15-902c-c2e0d45eed52","Type":"ContainerStarted","Data":"1aba5411c9c58a0f8d8a8e25d2ed661938822ff93a07d3945da37ffc73878990"} Oct 30 00:55:32.448174 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.448151 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" event={"ID":"08147324-90f2-4d15-902c-c2e0d45eed52","Type":"ContainerStarted","Data":"f22b7317c18571927f86c443e4722ffc1aabb474539aa66d16e7bb4f43e0d74d"} Oct 30 00:55:32.478649 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.478573 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:32.479953 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.479923 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:32.480208 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.480171 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:32.499483 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.499431 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podStartSLOduration=2.499413977 podStartE2EDuration="2.499413977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:32.497023575 +0000 UTC m=+1017.440196768" watchObservedRunningTime="2026-10-30 00:55:32.499413977 +0000 UTC m=+1017.442587171" Oct 30 00:55:32.592480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.592415 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podStartSLOduration=2.59239102 podStartE2EDuration="2.59239102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:32.552040274 +0000 UTC m=+1017.495213478" watchObservedRunningTime="2026-10-30 00:55:32.59239102 +0000 UTC m=+1017.535564212" Oct 30 00:55:32.998905 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.998874 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:32.999256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.999236 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 00:55:32.999477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:32.999452 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 00:55:33.480184 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:33.480091 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:33.480763 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:33.480731 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:33.994537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:33.994349 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 00:55:33.994537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:33.994415 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 00:55:34.387499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.387392 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76"] Oct 30 00:55:34.392985 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.392962 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.399126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.399101 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76"] Oct 30 00:55:34.409279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.409264 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:55:34.409461 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.409450 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:55:34.409621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.409556 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:55:34.409692 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.409644 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:55:34.437543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.433408 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.437543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.433616 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.437543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.433696 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rlbm\" (UniqueName: \"kubernetes.io/projected/91b45434-400b-41cd-8843-d10eb66b5154-kube-api-access-9rlbm\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.479749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.478946 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:34.479749 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.479004 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:34.558505 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.558471 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.573095 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.570620 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.573095 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.570709 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rlbm\" (UniqueName: \"kubernetes.io/projected/91b45434-400b-41cd-8843-d10eb66b5154-kube-api-access-9rlbm\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.573095 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.559746 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.588544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.587822 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.592000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.591976 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rlbm\" (UniqueName: \"kubernetes.io/projected/91b45434-400b-41cd-8843-d10eb66b5154-kube-api-access-9rlbm\") pod \"maas-api-revoke-sub-af7500e0-kxc76\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.708748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.708675 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:34.996143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.994960 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 00:55:34.996143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:34.995003 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 00:55:35.095755 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:35.094737 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76"] Oct 30 00:55:35.477643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:35.476039 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:35.477643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:35.476092 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:35.511614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:35.511574 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" event={"ID":"91b45434-400b-41cd-8843-d10eb66b5154","Type":"ContainerStarted","Data":"a82bf4d855a552232302d4f37615a47e8fa01aec4c97383e50ff0525876aa7da"} Oct 30 00:55:35.994976 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:35.994932 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 00:55:35.995586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:35.994986 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 00:55:36.483541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:36.482982 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:36.483541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:36.483042 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:36.522124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:36.520673 4905 generic.go:356] "Generic (PLEG): container finished" podID="91b45434-400b-41cd-8843-d10eb66b5154" containerID="b77e3ef59d754da11577b4ff949aeb1b439122b76020ae623085cee4cc131368" exitCode=0 Oct 30 00:55:36.522124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:36.520836 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" event={"ID":"91b45434-400b-41cd-8843-d10eb66b5154","Type":"ContainerDied","Data":"b77e3ef59d754da11577b4ff949aeb1b439122b76020ae623085cee4cc131368"} Oct 30 00:55:36.994850 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:36.994815 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 00:55:36.995024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:36.994873 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 00:55:37.304614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.303701 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5"] Oct 30 00:55:37.309081 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.309058 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.347123 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.347090 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5"] Oct 30 00:55:37.396451 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.396410 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.396851 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.396824 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.397160 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.397140 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncc2k\" (UniqueName: \"kubernetes.io/projected/8655488f-e05c-4647-95dd-c20d4703aee4-kube-api-access-ncc2k\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.491550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.489619 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:37.491550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.489682 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:37.510682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.504474 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncc2k\" (UniqueName: \"kubernetes.io/projected/8655488f-e05c-4647-95dd-c20d4703aee4-kube-api-access-ncc2k\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.510682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.504590 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.510682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.504753 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.510682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.506324 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.539573 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.534455 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.542620 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.537378 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncc2k\" (UniqueName: \"kubernetes.io/projected/8655488f-e05c-4647-95dd-c20d4703aee4-kube-api-access-ncc2k\") pod \"maas-api-revoke-sub-5696c81b-s8ff5\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.690026 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.689989 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:37.916364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.910805 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:37.995136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.995107 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 00:55:37.995639 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:37.995149 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 00:55:38.024263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.021626 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b45434-400b-41cd-8843-d10eb66b5154-kube-api-access-9rlbm\" (UniqueName: \"kubernetes.io/projected/91b45434-400b-41cd-8843-d10eb66b5154-kube-api-access-9rlbm\") pod \"91b45434-400b-41cd-8843-d10eb66b5154\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " Oct 30 00:55:38.024263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.021755 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-xks-ca\") pod \"91b45434-400b-41cd-8843-d10eb66b5154\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " Oct 30 00:55:38.024263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.021946 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-openshift-ca\") pod \"91b45434-400b-41cd-8843-d10eb66b5154\" (UID: \"91b45434-400b-41cd-8843-d10eb66b5154\") " Oct 30 00:55:38.024560 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.024437 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-openshift-ca" pod "91b45434-400b-41cd-8843-d10eb66b5154" (UID: "91b45434-400b-41cd-8843-d10eb66b5154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:38.068665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.068632 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5"] Oct 30 00:55:38.085280 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.085237 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b45434-400b-41cd-8843-d10eb66b5154-kube-api-access-9rlbm" pod "91b45434-400b-41cd-8843-d10eb66b5154" (UID: "91b45434-400b-41cd-8843-d10eb66b5154"). InnerVolumeSpecName "kube-api-access-9rlbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:38.093655 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:38.090045 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8655488f_e05c_4647_95dd_c20d4703aee4.slice/crio-bcfde9751af3e23911159c8e39a95edf2038f1eed7c15d20ab6d125c928e1e00 WatchSource:0}: Error finding container bcfde9751af3e23911159c8e39a95edf2038f1eed7c15d20ab6d125c928e1e00: Status 404 returned error can't find the container with id bcfde9751af3e23911159c8e39a95edf2038f1eed7c15d20ab6d125c928e1e00 Oct 30 00:55:38.093655 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.091735 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-xks-ca" pod "91b45434-400b-41cd-8843-d10eb66b5154" (UID: "91b45434-400b-41cd-8843-d10eb66b5154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:55:38.124300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.124250 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:38.124300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.124274 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rlbm\" (UniqueName: \"kubernetes.io/projected/91b45434-400b-41cd-8843-d10eb66b5154-kube-api-access-9rlbm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:38.124300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.124284 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45434-400b-41cd-8843-d10eb66b5154-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:38.476103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.475710 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:38.476103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.475784 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:38.604032 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.603999 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" event={"ID":"91b45434-400b-41cd-8843-d10eb66b5154","Type":"ContainerDied","Data":"a82bf4d855a552232302d4f37615a47e8fa01aec4c97383e50ff0525876aa7da"} Oct 30 00:55:38.604212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.604054 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a82bf4d855a552232302d4f37615a47e8fa01aec4c97383e50ff0525876aa7da" Oct 30 00:55:38.604212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.604012 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76" Oct 30 00:55:38.605843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.605816 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" event={"ID":"8655488f-e05c-4647-95dd-c20d4703aee4","Type":"ContainerStarted","Data":"b4d01791fd2fb5e5058ea4491615e8a7737956bcde441ff58f5737c784d33b77"} Oct 30 00:55:38.605975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.605860 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" event={"ID":"8655488f-e05c-4647-95dd-c20d4703aee4","Type":"ContainerStarted","Data":"bcfde9751af3e23911159c8e39a95edf2038f1eed7c15d20ab6d125c928e1e00"} Oct 30 00:55:38.656731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:38.656672 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" podStartSLOduration=1.656653957 podStartE2EDuration="1.656653957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:38.645810713 +0000 UTC m=+1023.588983908" watchObservedRunningTime="2026-10-30 00:55:38.656653957 +0000 UTC m=+1023.599827156" Oct 30 00:55:39.476689 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.476591 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:55:39.477219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.477185 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:55:39.503161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.503031 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:39.504123 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.503613 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:39.505798 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.505768 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 00:55:39.622790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.622758 4905 generic.go:356] "Generic (PLEG): container finished" podID="8655488f-e05c-4647-95dd-c20d4703aee4" containerID="b4d01791fd2fb5e5058ea4491615e8a7737956bcde441ff58f5737c784d33b77" exitCode=0 Oct 30 00:55:39.622971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.622829 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" event={"ID":"8655488f-e05c-4647-95dd-c20d4703aee4","Type":"ContainerDied","Data":"b4d01791fd2fb5e5058ea4491615e8a7737956bcde441ff58f5737c784d33b77"} Oct 30 00:55:39.793175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.793138 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7"] Oct 30 00:55:39.794093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.794071 4905 cpu_manager.go:414] "containerMap: removing container" podUID="91b45434-400b-41cd-8843-d10eb66b5154" containerName="revoke-keys" Oct 30 00:55:39.794191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.794098 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b45434-400b-41cd-8843-d10eb66b5154" containerName="revoke-keys" Oct 30 00:55:39.794327 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.794314 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b45434-400b-41cd-8843-d10eb66b5154" containerName="revoke-keys" Oct 30 00:55:39.798947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.798644 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:39.817299 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.817276 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7"] Oct 30 00:55:39.912395 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.912364 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckf74\" (UniqueName: \"kubernetes.io/projected/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-kube-api-access-ckf74\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:39.912594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.912568 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:39.912797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:39.912773 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.014105 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.014076 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.014276 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.014216 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.014392 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.014372 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckf74\" (UniqueName: \"kubernetes.io/projected/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-kube-api-access-ckf74\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.014983 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.014962 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.016766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.016746 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.034656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.034633 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckf74\" (UniqueName: \"kubernetes.io/projected/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-kube-api-access-ckf74\") pod \"maas-api-revoke-sub-dd33ef9b-4qpz7\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.119513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.119428 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:40.432736 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:40.432703 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7918536_7d0d_4835_a8f2_f2ef416c7b7b.slice/crio-4afbd8208715712c9bc8953f9acc9e372a2e7030a4494afdbccfe6b1272f899a WatchSource:0}: Error finding container 4afbd8208715712c9bc8953f9acc9e372a2e7030a4494afdbccfe6b1272f899a: Status 404 returned error can't find the container with id 4afbd8208715712c9bc8953f9acc9e372a2e7030a4494afdbccfe6b1272f899a Oct 30 00:55:40.434871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.434817 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7"] Oct 30 00:55:40.479795 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.479774 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:40.480203 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.480183 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:40.480891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.480874 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 00:55:40.632276 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.632245 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" event={"ID":"b7918536-7d0d-4835-a8f2-f2ef416c7b7b","Type":"ContainerStarted","Data":"4afbd8208715712c9bc8953f9acc9e372a2e7030a4494afdbccfe6b1272f899a"} Oct 30 00:55:40.788285 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.788262 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:40.928148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.927334 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8655488f-e05c-4647-95dd-c20d4703aee4-kube-api-access-ncc2k\" (UniqueName: \"kubernetes.io/projected/8655488f-e05c-4647-95dd-c20d4703aee4-kube-api-access-ncc2k\") pod \"8655488f-e05c-4647-95dd-c20d4703aee4\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " Oct 30 00:55:40.928148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.927476 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-xks-ca\") pod \"8655488f-e05c-4647-95dd-c20d4703aee4\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " Oct 30 00:55:40.928148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.927608 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-openshift-ca\") pod \"8655488f-e05c-4647-95dd-c20d4703aee4\" (UID: \"8655488f-e05c-4647-95dd-c20d4703aee4\") " Oct 30 00:55:40.928148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.928097 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-openshift-ca" pod "8655488f-e05c-4647-95dd-c20d4703aee4" (UID: "8655488f-e05c-4647-95dd-c20d4703aee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:40.928485 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.928386 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:40.930872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.930842 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-xks-ca" pod "8655488f-e05c-4647-95dd-c20d4703aee4" (UID: "8655488f-e05c-4647-95dd-c20d4703aee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:55:40.932499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:40.932473 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8655488f-e05c-4647-95dd-c20d4703aee4-kube-api-access-ncc2k" pod "8655488f-e05c-4647-95dd-c20d4703aee4" (UID: "8655488f-e05c-4647-95dd-c20d4703aee4"). InnerVolumeSpecName "kube-api-access-ncc2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:41.035905 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:41.034054 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncc2k\" (UniqueName: \"kubernetes.io/projected/8655488f-e05c-4647-95dd-c20d4703aee4-kube-api-access-ncc2k\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:41.035905 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:41.034092 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8655488f-e05c-4647-95dd-c20d4703aee4-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:41.644550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:41.644501 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" Oct 30 00:55:41.644992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:41.644541 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5" event={"ID":"8655488f-e05c-4647-95dd-c20d4703aee4","Type":"ContainerDied","Data":"bcfde9751af3e23911159c8e39a95edf2038f1eed7c15d20ab6d125c928e1e00"} Oct 30 00:55:41.644992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:41.644611 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcfde9751af3e23911159c8e39a95edf2038f1eed7c15d20ab6d125c928e1e00" Oct 30 00:55:41.646686 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:41.646659 4905 generic.go:356] "Generic (PLEG): container finished" podID="b7918536-7d0d-4835-a8f2-f2ef416c7b7b" containerID="35463962c0f930aab1da36b26f19575f8dfc535e2884c0a4ad58ecaa8dfc6cd6" exitCode=0 Oct 30 00:55:41.646846 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:41.646733 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" event={"ID":"b7918536-7d0d-4835-a8f2-f2ef416c7b7b","Type":"ContainerDied","Data":"35463962c0f930aab1da36b26f19575f8dfc535e2884c0a4ad58ecaa8dfc6cd6"} Oct 30 00:55:42.415321 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:55:42.415287 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:55:42.818047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.818022 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:42.959291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.959263 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-openshift-ca\") pod \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " Oct 30 00:55:42.959492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.959401 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-kube-api-access-ckf74\" (UniqueName: \"kubernetes.io/projected/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-kube-api-access-ckf74\") pod \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " Oct 30 00:55:42.959584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.959565 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-xks-ca\") pod \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\" (UID: \"b7918536-7d0d-4835-a8f2-f2ef416c7b7b\") " Oct 30 00:55:42.959671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.959644 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-openshift-ca" pod "b7918536-7d0d-4835-a8f2-f2ef416c7b7b" (UID: "b7918536-7d0d-4835-a8f2-f2ef416c7b7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:42.960124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.960062 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.961656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.961633 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-kube-api-access-ckf74" pod "b7918536-7d0d-4835-a8f2-f2ef416c7b7b" (UID: "b7918536-7d0d-4835-a8f2-f2ef416c7b7b"). InnerVolumeSpecName "kube-api-access-ckf74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:42.962100 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:42.962068 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-xks-ca" pod "b7918536-7d0d-4835-a8f2-f2ef416c7b7b" (UID: "b7918536-7d0d-4835-a8f2-f2ef416c7b7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:55:43.061279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:43.061244 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckf74\" (UniqueName: \"kubernetes.io/projected/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-kube-api-access-ckf74\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:43.061279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:43.061271 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7918536-7d0d-4835-a8f2-f2ef416c7b7b-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:43.664549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:43.664491 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" Oct 30 00:55:43.664722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:43.664484 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7" event={"ID":"b7918536-7d0d-4835-a8f2-f2ef416c7b7b","Type":"ContainerDied","Data":"4afbd8208715712c9bc8953f9acc9e372a2e7030a4494afdbccfe6b1272f899a"} Oct 30 00:55:43.664722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:43.664638 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afbd8208715712c9bc8953f9acc9e372a2e7030a4494afdbccfe6b1272f899a" Oct 30 00:55:46.007719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.007381 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd"] Oct 30 00:55:46.055551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.037461 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b7918536-7d0d-4835-a8f2-f2ef416c7b7b" containerName="revoke-keys" Oct 30 00:55:46.055816 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.055787 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7918536-7d0d-4835-a8f2-f2ef416c7b7b" containerName="revoke-keys" Oct 30 00:55:46.056021 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.056003 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8655488f-e05c-4647-95dd-c20d4703aee4" containerName="revoke-keys" Oct 30 00:55:46.064599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.064569 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8655488f-e05c-4647-95dd-c20d4703aee4" containerName="revoke-keys" Oct 30 00:55:46.076550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.064983 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7918536-7d0d-4835-a8f2-f2ef416c7b7b" containerName="revoke-keys" Oct 30 00:55:46.076784 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.076760 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8655488f-e05c-4647-95dd-c20d4703aee4" containerName="revoke-keys" Oct 30 00:55:46.256914 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.256883 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.358637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.358600 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.358929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.358890 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.359185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.359164 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.359389 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.359357 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22737c92-facc-49a0-b4a2-4980aacd0c36-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.359662 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.359641 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.359864 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.359846 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.360042 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.360025 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.360227 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.360210 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.360480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.360461 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4b5\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-kube-api-access-gg4b5\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.389456 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.389427 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd"] Oct 30 00:55:46.482646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.482607 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.490757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.490729 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.491094 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.491064 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.491330 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.491312 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.491545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.491510 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4b5\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-kube-api-access-gg4b5\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.491739 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.491706 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.491847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.491829 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.511211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.491861 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.511211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.504028 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.511211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.504097 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22737c92-facc-49a0-b4a2-4980aacd0c36-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.511735 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.503955 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.512215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.483571 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.533542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.531898 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22737c92-facc-49a0-b4a2-4980aacd0c36-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.533542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.532409 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.533542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.532675 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.535650 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.535626 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.563583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.560583 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4b5\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-kube-api-access-gg4b5\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.573541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.569581 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.679566 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.674780 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:46.934645 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.933597 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:46.956916 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:46.955628 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:47.010541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.008593 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:47.224637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.223630 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:55:47.528182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.528142 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:47.612595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.612562 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt"] Oct 30 00:55:47.689005 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.688958 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.706160 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.706115 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt"] Oct 30 00:55:47.708852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.708401 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:55:47.708852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.708581 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:55:47.708852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.708628 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:55:47.716553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.708715 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:55:47.746274 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.746244 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.746499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.746481 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zql\" (UniqueName: \"kubernetes.io/projected/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-kube-api-access-d6zql\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.746752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.746721 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.829346 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.825544 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:47.833211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.832673 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:47.833211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.832762 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:47.833211 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.832801 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:55:47.856466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.844016 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd"] Oct 30 00:55:47.856466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.848737 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.856466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.849028 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.856466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.849097 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zql\" (UniqueName: \"kubernetes.io/projected/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-kube-api-access-d6zql\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.856466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.850695 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.860291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.858750 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" event={"ID":"22737c92-facc-49a0-b4a2-4980aacd0c36","Type":"ContainerStarted","Data":"4f6e67db69a6e0a00eda25d50301313fb0d6af544cca69d7497507f54d9c851f"} Oct 30 00:55:47.860291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.860186 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.881129 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.872571 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zql\" (UniqueName: \"kubernetes.io/projected/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-kube-api-access-d6zql\") pod \"maas-api-revoke-sub-cfacb692-gvgqt\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:47.907543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:47.905561 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:48.024969 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.024929 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:48.532595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.531620 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b"] Oct 30 00:55:48.566927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.557773 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.566927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.560372 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b"] Oct 30 00:55:48.612974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.612945 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.613099 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.612980 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.613159 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.613147 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cj6v\" (UniqueName: \"kubernetes.io/projected/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-kube-api-access-9cj6v\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.714277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.714246 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cj6v\" (UniqueName: \"kubernetes.io/projected/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-kube-api-access-9cj6v\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.714433 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.714417 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.714472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.714448 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.715330 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.715304 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.716847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.716825 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.734877 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.734846 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cj6v\" (UniqueName: \"kubernetes.io/projected/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-kube-api-access-9cj6v\") pod \"maas-api-revoke-sub-f1609a64-xcx5b\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.763657 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:48.763629 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cbb50b_8b3e_4f5a_8c44_4a8640c66e4f.slice/crio-bb0eeaf6390700afa0337b6704581b13e812096fccd1303f7fd8a12696fb4bf4 WatchSource:0}: Error finding container bb0eeaf6390700afa0337b6704581b13e812096fccd1303f7fd8a12696fb4bf4: Status 404 returned error can't find the container with id bb0eeaf6390700afa0337b6704581b13e812096fccd1303f7fd8a12696fb4bf4 Oct 30 00:55:48.766843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.766821 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt"] Oct 30 00:55:48.869509 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.869473 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" event={"ID":"22737c92-facc-49a0-b4a2-4980aacd0c36","Type":"ContainerStarted","Data":"1664e06f01c5c5f3275a7d7a7f67ae122889c5dca64e8f4442ad29c836e5b7c8"} Oct 30 00:55:48.871189 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.871147 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" event={"ID":"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f","Type":"ContainerStarted","Data":"bb0eeaf6390700afa0337b6704581b13e812096fccd1303f7fd8a12696fb4bf4"} Oct 30 00:55:48.889457 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.889439 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:48.916283 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:48.916242 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podStartSLOduration=3.916229732 podStartE2EDuration="3.916229732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:48.913429504 +0000 UTC m=+1033.856602709" watchObservedRunningTime="2026-10-30 00:55:48.916229732 +0000 UTC m=+1033.859402923" Oct 30 00:55:49.202127 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:49.200836 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8754e4f1_dcf3_44d0_a04d_09b44b421a9f.slice/crio-44d0b95c00f0c3228496dbb1f2fb9329ac751f63bb87d979448c814e7a660757 WatchSource:0}: Error finding container 44d0b95c00f0c3228496dbb1f2fb9329ac751f63bb87d979448c814e7a660757: Status 404 returned error can't find the container with id 44d0b95c00f0c3228496dbb1f2fb9329ac751f63bb87d979448c814e7a660757 Oct 30 00:55:49.203485 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.203286 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b"] Oct 30 00:55:49.675351 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.675323 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:49.676328 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.676303 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:55:49.676415 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.676353 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:55:49.880788 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.880752 4905 generic.go:356] "Generic (PLEG): container finished" podID="26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" containerID="9264f35003b9a122b9c55f2544ece478d975fcb79ede8d8d6bb5f5840d99a990" exitCode=0 Oct 30 00:55:49.880977 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.880842 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" event={"ID":"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f","Type":"ContainerDied","Data":"9264f35003b9a122b9c55f2544ece478d975fcb79ede8d8d6bb5f5840d99a990"} Oct 30 00:55:49.882574 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.882544 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" event={"ID":"8754e4f1-dcf3-44d0-a04d-09b44b421a9f","Type":"ContainerStarted","Data":"b052c851b2f0eea470d02be7d687c454cab9ef5129a8551456f079f626aecdcf"} Oct 30 00:55:49.882721 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.882590 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" event={"ID":"8754e4f1-dcf3-44d0-a04d-09b44b421a9f","Type":"ContainerStarted","Data":"44d0b95c00f0c3228496dbb1f2fb9329ac751f63bb87d979448c814e7a660757"} Oct 30 00:55:49.966541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:49.966440 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" podStartSLOduration=1.966422178 podStartE2EDuration="1.966422178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:49.965845731 +0000 UTC m=+1034.909018925" watchObservedRunningTime="2026-10-30 00:55:49.966422178 +0000 UTC m=+1034.909595372" Oct 30 00:55:50.676507 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:50.676406 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:55:50.676507 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:50.676464 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:55:50.891955 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:50.891923 4905 generic.go:356] "Generic (PLEG): container finished" podID="8754e4f1-dcf3-44d0-a04d-09b44b421a9f" containerID="b052c851b2f0eea470d02be7d687c454cab9ef5129a8551456f079f626aecdcf" exitCode=0 Oct 30 00:55:50.892114 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:50.892012 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" event={"ID":"8754e4f1-dcf3-44d0-a04d-09b44b421a9f","Type":"ContainerDied","Data":"b052c851b2f0eea470d02be7d687c454cab9ef5129a8551456f079f626aecdcf"} Oct 30 00:55:51.165903 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.165873 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:51.202717 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.202687 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-xks-ca\") pod \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " Oct 30 00:55:51.202852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.202751 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-kube-api-access-d6zql\" (UniqueName: \"kubernetes.io/projected/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-kube-api-access-d6zql\") pod \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " Oct 30 00:55:51.203072 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.203052 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-openshift-ca\") pod \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\" (UID: \"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f\") " Oct 30 00:55:51.203824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.203784 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-openshift-ca" pod "26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" (UID: "26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:51.212277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.204952 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-kube-api-access-d6zql" pod "26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" (UID: "26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f"). InnerVolumeSpecName "kube-api-access-d6zql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:51.224175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.224146 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-xks-ca" pod "26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" (UID: "26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:55:51.306037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.306017 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:51.306174 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.306165 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:51.306227 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.306219 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zql\" (UniqueName: \"kubernetes.io/projected/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f-kube-api-access-d6zql\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:51.676126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.676040 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:55:51.676126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.676093 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:55:51.864549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.861975 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk"] Oct 30 00:55:51.864549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.863086 4905 cpu_manager.go:414] "containerMap: removing container" podUID="26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" containerName="revoke-keys" Oct 30 00:55:51.864549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.863107 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" containerName="revoke-keys" Oct 30 00:55:51.864549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.863311 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" containerName="revoke-keys" Oct 30 00:55:51.885182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.874963 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:51.932586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.931737 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:51.932586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.931852 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:51.932586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.931877 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtg8\" (UniqueName: \"kubernetes.io/projected/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-kube-api-access-cwtg8\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:51.964577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.957886 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk"] Oct 30 00:55:51.978430 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.978396 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" Oct 30 00:55:51.978638 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.978443 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt" event={"ID":"26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f","Type":"ContainerDied","Data":"bb0eeaf6390700afa0337b6704581b13e812096fccd1303f7fd8a12696fb4bf4"} Oct 30 00:55:51.978638 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:51.978486 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb0eeaf6390700afa0337b6704581b13e812096fccd1303f7fd8a12696fb4bf4" Oct 30 00:55:52.061559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.057794 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:52.061559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.057958 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:52.061559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.057991 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtg8\" (UniqueName: \"kubernetes.io/projected/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-kube-api-access-cwtg8\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:52.061559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.061229 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:52.066083 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.063929 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:52.086872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.086848 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtg8\" (UniqueName: \"kubernetes.io/projected/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-kube-api-access-cwtg8\") pod \"maas-api-revoke-sub-9b44830f-nw5zk\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:52.210496 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.210464 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:52.274124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.274099 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:52.369319 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.368183 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-kube-api-access-9cj6v\" (UniqueName: \"kubernetes.io/projected/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-kube-api-access-9cj6v\") pod \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " Oct 30 00:55:52.369319 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.368254 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-xks-ca\") pod \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " Oct 30 00:55:52.369319 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.368333 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-openshift-ca\") pod \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\" (UID: \"8754e4f1-dcf3-44d0-a04d-09b44b421a9f\") " Oct 30 00:55:52.369618 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.369389 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-openshift-ca" pod "8754e4f1-dcf3-44d0-a04d-09b44b421a9f" (UID: "8754e4f1-dcf3-44d0-a04d-09b44b421a9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:52.379124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.378678 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-kube-api-access-9cj6v" pod "8754e4f1-dcf3-44d0-a04d-09b44b421a9f" (UID: "8754e4f1-dcf3-44d0-a04d-09b44b421a9f"). InnerVolumeSpecName "kube-api-access-9cj6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:52.383608 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.382664 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-xks-ca" pod "8754e4f1-dcf3-44d0-a04d-09b44b421a9f" (UID: "8754e4f1-dcf3-44d0-a04d-09b44b421a9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:55:52.471580 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.471386 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cj6v\" (UniqueName: \"kubernetes.io/projected/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-kube-api-access-9cj6v\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:52.471580 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.471414 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:52.471580 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.471429 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8754e4f1-dcf3-44d0-a04d-09b44b421a9f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:52.615733 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:55:52.615701 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8078c6c1_0bc7_40ac_91d9_d23f84f4d5de.slice/crio-6eb913c25138c84b484b232c3b9481236c2d2c2a8e3237b06127004d23ed5472 WatchSource:0}: Error finding container 6eb913c25138c84b484b232c3b9481236c2d2c2a8e3237b06127004d23ed5472: Status 404 returned error can't find the container with id 6eb913c25138c84b484b232c3b9481236c2d2c2a8e3237b06127004d23ed5472 Oct 30 00:55:52.617154 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.617125 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk"] Oct 30 00:55:52.676075 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.676038 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:55:52.676180 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.676098 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:55:52.997185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.997161 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" Oct 30 00:55:52.997714 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.997155 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b" event={"ID":"8754e4f1-dcf3-44d0-a04d-09b44b421a9f","Type":"ContainerDied","Data":"44d0b95c00f0c3228496dbb1f2fb9329ac751f63bb87d979448c814e7a660757"} Oct 30 00:55:52.997714 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.997307 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44d0b95c00f0c3228496dbb1f2fb9329ac751f63bb87d979448c814e7a660757" Oct 30 00:55:52.998649 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:52.998623 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" event={"ID":"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de","Type":"ContainerStarted","Data":"6eb913c25138c84b484b232c3b9481236c2d2c2a8e3237b06127004d23ed5472"} Oct 30 00:55:53.676372 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:53.676337 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:55:53.676690 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:53.676656 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:55:54.007208 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:54.007181 4905 generic.go:356] "Generic (PLEG): container finished" podID="8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" containerID="d168272ec28d9bdbfc6ff36c35493cbfb6abb7ca67c245f69b8c8cf558e9e624" exitCode=0 Oct 30 00:55:54.007625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:54.007227 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" event={"ID":"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de","Type":"ContainerDied","Data":"d168272ec28d9bdbfc6ff36c35493cbfb6abb7ca67c245f69b8c8cf558e9e624"} Oct 30 00:55:54.675890 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:54.675853 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:55:54.676080 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:54.675917 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:55:55.342703 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.342677 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:55.467938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.467903 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-openshift-ca\") pod \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " Oct 30 00:55:55.468098 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.467951 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-xks-ca\") pod \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " Oct 30 00:55:55.468179 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.468101 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-kube-api-access-cwtg8\" (UniqueName: \"kubernetes.io/projected/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-kube-api-access-cwtg8\") pod \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\" (UID: \"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de\") " Oct 30 00:55:55.468358 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.468332 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-openshift-ca" pod "8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" (UID: "8078c6c1-0bc7-40ac-91d9-d23f84f4d5de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:55.468820 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.468741 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:55.485066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.484724 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-xks-ca" pod "8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" (UID: "8078c6c1-0bc7-40ac-91d9-d23f84f4d5de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:55:55.485066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.484890 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-kube-api-access-cwtg8" pod "8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" (UID: "8078c6c1-0bc7-40ac-91d9-d23f84f4d5de"). InnerVolumeSpecName "kube-api-access-cwtg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:55.569687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.569616 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtg8\" (UniqueName: \"kubernetes.io/projected/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-kube-api-access-cwtg8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:55.569687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:55.569639 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:55:56.065998 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:56.065964 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" Oct 30 00:55:56.065998 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:56.065992 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk" event={"ID":"8078c6c1-0bc7-40ac-91d9-d23f84f4d5de","Type":"ContainerDied","Data":"6eb913c25138c84b484b232c3b9481236c2d2c2a8e3237b06127004d23ed5472"} Oct 30 00:55:56.066282 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:56.066036 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb913c25138c84b484b232c3b9481236c2d2c2a8e3237b06127004d23ed5472" Oct 30 00:55:56.078572 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:56.078110 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:56.078698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:56.078681 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:55:56.081042 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:55:56.081022 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 00:56:04.898221 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.898190 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc"] Oct 30 00:56:04.898827 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.898813 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" containerName="revoke-keys" Oct 30 00:56:04.898869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.898829 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" containerName="revoke-keys" Oct 30 00:56:04.898869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.898841 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8754e4f1-dcf3-44d0-a04d-09b44b421a9f" containerName="revoke-keys" Oct 30 00:56:04.898869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.898846 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8754e4f1-dcf3-44d0-a04d-09b44b421a9f" containerName="revoke-keys" Oct 30 00:56:04.898988 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.898978 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" containerName="revoke-keys" Oct 30 00:56:04.899026 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.898989 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8754e4f1-dcf3-44d0-a04d-09b44b421a9f" containerName="revoke-keys" Oct 30 00:56:04.903983 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.903963 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:04.920128 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.920106 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc"] Oct 30 00:56:04.931296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.929534 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:56:04.931296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.929571 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:56:04.931296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.929610 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:56:04.933795 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.933773 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:56:04.944408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.944369 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqt8\" (UniqueName: \"kubernetes.io/projected/8211c053-91f4-4864-8883-2d08931aa5c9-kube-api-access-pkqt8\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:04.944565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.944451 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:04.944629 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:04.944602 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.046005 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.045969 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqt8\" (UniqueName: \"kubernetes.io/projected/8211c053-91f4-4864-8883-2d08931aa5c9-kube-api-access-pkqt8\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.046167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.046056 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.046235 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.046187 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.046879 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.046856 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.048634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.048611 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.066855 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.066823 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqt8\" (UniqueName: \"kubernetes.io/projected/8211c053-91f4-4864-8883-2d08931aa5c9-kube-api-access-pkqt8\") pod \"maas-api-revoke-sub-c3e9a128-j6hqc\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.218115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.218039 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:05.519189 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:05.519159 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8211c053_91f4_4864_8883_2d08931aa5c9.slice/crio-6cb8ae3eb6b0979c526e2fd0229cc2fe386c4342fc1e35eaa072653a85a9e818 WatchSource:0}: Error finding container 6cb8ae3eb6b0979c526e2fd0229cc2fe386c4342fc1e35eaa072653a85a9e818: Status 404 returned error can't find the container with id 6cb8ae3eb6b0979c526e2fd0229cc2fe386c4342fc1e35eaa072653a85a9e818 Oct 30 00:56:05.521586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.521555 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc"] Oct 30 00:56:05.576164 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.576131 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6"] Oct 30 00:56:05.581778 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.581755 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.599552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.599505 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6"] Oct 30 00:56:05.654783 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.654753 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.654917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.654810 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.655069 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.655032 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6fg4\" (UniqueName: \"kubernetes.io/projected/6d3a486f-8f39-4cf0-8113-aacea94f695b-kube-api-access-t6fg4\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.756756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.756719 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.756904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.756773 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.756981 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.756932 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6fg4\" (UniqueName: \"kubernetes.io/projected/6d3a486f-8f39-4cf0-8113-aacea94f695b-kube-api-access-t6fg4\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.757417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.757395 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.759322 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.759297 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.777354 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.777291 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6fg4\" (UniqueName: \"kubernetes.io/projected/6d3a486f-8f39-4cf0-8113-aacea94f695b-kube-api-access-t6fg4\") pod \"maas-api-revoke-sub-0976d365-l6ff6\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:05.894909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:05.894877 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:06.148255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:06.148230 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" event={"ID":"8211c053-91f4-4864-8883-2d08931aa5c9","Type":"ContainerStarted","Data":"db724e2c9bbe62f30825e2e8c501b628d38358ae2cc23bfb44eaa9408b21e094"} Oct 30 00:56:06.148687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:06.148294 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" event={"ID":"8211c053-91f4-4864-8883-2d08931aa5c9","Type":"ContainerStarted","Data":"6cb8ae3eb6b0979c526e2fd0229cc2fe386c4342fc1e35eaa072653a85a9e818"} Oct 30 00:56:06.214543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:06.212898 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" podStartSLOduration=2.212878941 podStartE2EDuration="2.212878941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:06.210948946 +0000 UTC m=+1051.154122140" watchObservedRunningTime="2026-10-30 00:56:06.212878941 +0000 UTC m=+1051.156052133" Oct 30 00:56:06.283846 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:06.283745 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6"] Oct 30 00:56:07.256675 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:07.256628 4905 generic.go:356] "Generic (PLEG): container finished" podID="8211c053-91f4-4864-8883-2d08931aa5c9" containerID="db724e2c9bbe62f30825e2e8c501b628d38358ae2cc23bfb44eaa9408b21e094" exitCode=0 Oct 30 00:56:07.289933 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:07.289896 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" event={"ID":"6d3a486f-8f39-4cf0-8113-aacea94f695b","Type":"ContainerStarted","Data":"d549673f49083d97c4718c0f9902a49495484fa90506d5f70375da43d61bd004"} Oct 30 00:56:07.290192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:07.290173 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" event={"ID":"6d3a486f-8f39-4cf0-8113-aacea94f695b","Type":"ContainerStarted","Data":"c81a6be7c64de4989754959208247641a1f2e8f74cf6cd3181ace5857db2641f"} Oct 30 00:56:07.290337 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:07.290323 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" event={"ID":"8211c053-91f4-4864-8883-2d08931aa5c9","Type":"ContainerDied","Data":"db724e2c9bbe62f30825e2e8c501b628d38358ae2cc23bfb44eaa9408b21e094"} Oct 30 00:56:07.304623 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:07.304568 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" podStartSLOduration=2.304550661 podStartE2EDuration="2.304550661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:07.302939217 +0000 UTC m=+1052.246112410" watchObservedRunningTime="2026-10-30 00:56:07.304550661 +0000 UTC m=+1052.247723868" Oct 30 00:56:08.297544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:08.297124 4905 generic.go:356] "Generic (PLEG): container finished" podID="6d3a486f-8f39-4cf0-8113-aacea94f695b" containerID="d549673f49083d97c4718c0f9902a49495484fa90506d5f70375da43d61bd004" exitCode=0 Oct 30 00:56:08.298076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:08.297976 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" event={"ID":"6d3a486f-8f39-4cf0-8113-aacea94f695b","Type":"ContainerDied","Data":"d549673f49083d97c4718c0f9902a49495484fa90506d5f70375da43d61bd004"} Oct 30 00:56:08.855545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:08.828619 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.195031 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.194930 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.216100 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.215706 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.235792 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.235430 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.590591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.543662 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:56:09.590591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.543728 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.590591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.545321 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.590591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.567613 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.699307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.699276 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:09.852543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.852099 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:09.858535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.853777 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:56:09.900919 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.900888 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-openshift-ca\") pod \"8211c053-91f4-4864-8883-2d08931aa5c9\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " Oct 30 00:56:09.901179 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.901160 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8211c053-91f4-4864-8883-2d08931aa5c9-kube-api-access-pkqt8\" (UniqueName: \"kubernetes.io/projected/8211c053-91f4-4864-8883-2d08931aa5c9-kube-api-access-pkqt8\") pod \"8211c053-91f4-4864-8883-2d08931aa5c9\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " Oct 30 00:56:09.901436 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.901414 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-xks-ca\") pod \"8211c053-91f4-4864-8883-2d08931aa5c9\" (UID: \"8211c053-91f4-4864-8883-2d08931aa5c9\") " Oct 30 00:56:09.907882 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:09.907857 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-openshift-ca" pod "8211c053-91f4-4864-8883-2d08931aa5c9" (UID: "8211c053-91f4-4864-8883-2d08931aa5c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:10.015501 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.009142 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:10.032589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.025184 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:10.224431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.221217 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-xks-ca" pod "8211c053-91f4-4864-8883-2d08931aa5c9" (UID: "8211c053-91f4-4864-8883-2d08931aa5c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:10.224431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.221332 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8211c053-91f4-4864-8883-2d08931aa5c9-kube-api-access-pkqt8" pod "8211c053-91f4-4864-8883-2d08931aa5c9" (UID: "8211c053-91f4-4864-8883-2d08931aa5c9"). InnerVolumeSpecName "kube-api-access-pkqt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:10.224431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.223197 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:56:10.233365 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.233340 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj"] Oct 30 00:56:10.236057 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.236037 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqt8\" (UniqueName: \"kubernetes.io/projected/8211c053-91f4-4864-8883-2d08931aa5c9-kube-api-access-pkqt8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:10.236241 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.236225 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8211c053-91f4-4864-8883-2d08931aa5c9-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:10.254318 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.254297 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8211c053-91f4-4864-8883-2d08931aa5c9" containerName="revoke-keys" Oct 30 00:56:10.254492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.254477 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8211c053-91f4-4864-8883-2d08931aa5c9" containerName="revoke-keys" Oct 30 00:56:10.255234 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.255126 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8211c053-91f4-4864-8883-2d08931aa5c9" containerName="revoke-keys" Oct 30 00:56:10.275198 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.275176 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.278049 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.278028 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj"] Oct 30 00:56:10.364459 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.364428 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.366023 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.365993 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dpw\" (UniqueName: \"kubernetes.io/projected/3c35ed98-93b1-4adc-8e86-565bce5af6ca-kube-api-access-22dpw\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.366155 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.366110 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.443142 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.443108 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:10.466927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.466891 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3a486f-8f39-4cf0-8113-aacea94f695b-kube-api-access-t6fg4\" (UniqueName: \"kubernetes.io/projected/6d3a486f-8f39-4cf0-8113-aacea94f695b-kube-api-access-t6fg4\") pod \"6d3a486f-8f39-4cf0-8113-aacea94f695b\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " Oct 30 00:56:10.467366 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.467341 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-openshift-ca\") pod \"6d3a486f-8f39-4cf0-8113-aacea94f695b\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " Oct 30 00:56:10.467591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.467571 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-xks-ca\") pod \"6d3a486f-8f39-4cf0-8113-aacea94f695b\" (UID: \"6d3a486f-8f39-4cf0-8113-aacea94f695b\") " Oct 30 00:56:10.467991 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.467957 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-openshift-ca" pod "6d3a486f-8f39-4cf0-8113-aacea94f695b" (UID: "6d3a486f-8f39-4cf0-8113-aacea94f695b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:10.468262 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.468238 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dpw\" (UniqueName: \"kubernetes.io/projected/3c35ed98-93b1-4adc-8e86-565bce5af6ca-kube-api-access-22dpw\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.468766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.468741 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.469399 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.469368 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.470149 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.470107 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.470476 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.470456 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:10.471148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.471117 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-xks-ca" pod "6d3a486f-8f39-4cf0-8113-aacea94f695b" (UID: "6d3a486f-8f39-4cf0-8113-aacea94f695b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:10.473434 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.473390 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.474104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.474070 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3a486f-8f39-4cf0-8113-aacea94f695b-kube-api-access-t6fg4" pod "6d3a486f-8f39-4cf0-8113-aacea94f695b" (UID: "6d3a486f-8f39-4cf0-8113-aacea94f695b"). InnerVolumeSpecName "kube-api-access-t6fg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:10.496175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.496119 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dpw\" (UniqueName: \"kubernetes.io/projected/3c35ed98-93b1-4adc-8e86-565bce5af6ca-kube-api-access-22dpw\") pod \"maas-api-revoke-sub-1253324e-txqnj\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.572341 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.572308 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6fg4\" (UniqueName: \"kubernetes.io/projected/6d3a486f-8f39-4cf0-8113-aacea94f695b-kube-api-access-t6fg4\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:10.572341 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.572336 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3a486f-8f39-4cf0-8113-aacea94f695b-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:10.668209 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.668177 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:10.679943 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.679915 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" Oct 30 00:56:10.680076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.679915 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc" event={"ID":"8211c053-91f4-4864-8883-2d08931aa5c9","Type":"ContainerDied","Data":"6cb8ae3eb6b0979c526e2fd0229cc2fe386c4342fc1e35eaa072653a85a9e818"} Oct 30 00:56:10.680122 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.680091 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cb8ae3eb6b0979c526e2fd0229cc2fe386c4342fc1e35eaa072653a85a9e818" Oct 30 00:56:10.681776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.681755 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" Oct 30 00:56:10.681889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.681795 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6" event={"ID":"6d3a486f-8f39-4cf0-8113-aacea94f695b","Type":"ContainerDied","Data":"c81a6be7c64de4989754959208247641a1f2e8f74cf6cd3181ace5857db2641f"} Oct 30 00:56:10.681889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.681841 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c81a6be7c64de4989754959208247641a1f2e8f74cf6cd3181ace5857db2641f" Oct 30 00:56:10.971467 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:10.971436 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c35ed98_93b1_4adc_8e86_565bce5af6ca.slice/crio-39ca6b6122b27759cf9c968f2f0770b1e447ee3dfe89c0cf54d2ed2c799377c7 WatchSource:0}: Error finding container 39ca6b6122b27759cf9c968f2f0770b1e447ee3dfe89c0cf54d2ed2c799377c7: Status 404 returned error can't find the container with id 39ca6b6122b27759cf9c968f2f0770b1e447ee3dfe89c0cf54d2ed2c799377c7 Oct 30 00:56:10.974832 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:10.974808 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj"] Oct 30 00:56:11.690918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:11.690824 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" event={"ID":"3c35ed98-93b1-4adc-8e86-565bce5af6ca","Type":"ContainerStarted","Data":"e13c4b0c400c1b5dcf7c3adf69d958b0f6c8ed1e134cc935f4ae2551fb9fbe70"} Oct 30 00:56:11.690918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:11.690881 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" event={"ID":"3c35ed98-93b1-4adc-8e86-565bce5af6ca","Type":"ContainerStarted","Data":"39ca6b6122b27759cf9c968f2f0770b1e447ee3dfe89c0cf54d2ed2c799377c7"} Oct 30 00:56:11.733597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:11.733551 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" podStartSLOduration=1.73353384 podStartE2EDuration="1.73353384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:11.730580931 +0000 UTC m=+1056.673754124" watchObservedRunningTime="2026-10-30 00:56:11.73353384 +0000 UTC m=+1056.676707024" Oct 30 00:56:12.414627 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:56:12.414592 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:56:12.699860 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:12.699779 4905 generic.go:356] "Generic (PLEG): container finished" podID="3c35ed98-93b1-4adc-8e86-565bce5af6ca" containerID="e13c4b0c400c1b5dcf7c3adf69d958b0f6c8ed1e134cc935f4ae2551fb9fbe70" exitCode=0 Oct 30 00:56:12.700238 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:12.699877 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" event={"ID":"3c35ed98-93b1-4adc-8e86-565bce5af6ca","Type":"ContainerDied","Data":"e13c4b0c400c1b5dcf7c3adf69d958b0f6c8ed1e134cc935f4ae2551fb9fbe70"} Oct 30 00:56:13.872015 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.871981 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:13.921874 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.921847 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-openshift-ca\") pod \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " Oct 30 00:56:13.922035 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.921926 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-xks-ca\") pod \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " Oct 30 00:56:13.922035 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.922015 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c35ed98-93b1-4adc-8e86-565bce5af6ca-kube-api-access-22dpw\" (UniqueName: \"kubernetes.io/projected/3c35ed98-93b1-4adc-8e86-565bce5af6ca-kube-api-access-22dpw\") pod \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\" (UID: \"3c35ed98-93b1-4adc-8e86-565bce5af6ca\") " Oct 30 00:56:13.922233 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.922212 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-openshift-ca" pod "3c35ed98-93b1-4adc-8e86-565bce5af6ca" (UID: "3c35ed98-93b1-4adc-8e86-565bce5af6ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:13.922839 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.922813 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:13.924426 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.924396 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35ed98-93b1-4adc-8e86-565bce5af6ca-kube-api-access-22dpw" pod "3c35ed98-93b1-4adc-8e86-565bce5af6ca" (UID: "3c35ed98-93b1-4adc-8e86-565bce5af6ca"). InnerVolumeSpecName "kube-api-access-22dpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:13.935626 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:13.935602 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-xks-ca" pod "3c35ed98-93b1-4adc-8e86-565bce5af6ca" (UID: "3c35ed98-93b1-4adc-8e86-565bce5af6ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:14.023759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:14.023734 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c35ed98-93b1-4adc-8e86-565bce5af6ca-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:14.023759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:14.023760 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22dpw\" (UniqueName: \"kubernetes.io/projected/3c35ed98-93b1-4adc-8e86-565bce5af6ca-kube-api-access-22dpw\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:14.724438 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:14.724404 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" event={"ID":"3c35ed98-93b1-4adc-8e86-565bce5af6ca","Type":"ContainerDied","Data":"39ca6b6122b27759cf9c968f2f0770b1e447ee3dfe89c0cf54d2ed2c799377c7"} Oct 30 00:56:14.724438 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:14.724424 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj" Oct 30 00:56:14.724720 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:14.724462 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ca6b6122b27759cf9c968f2f0770b1e447ee3dfe89c0cf54d2ed2c799377c7" Oct 30 00:56:15.887047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.887003 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4"] Oct 30 00:56:15.887853 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.887835 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6d3a486f-8f39-4cf0-8113-aacea94f695b" containerName="revoke-keys" Oct 30 00:56:15.887909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.887855 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3a486f-8f39-4cf0-8113-aacea94f695b" containerName="revoke-keys" Oct 30 00:56:15.887909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.887876 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3c35ed98-93b1-4adc-8e86-565bce5af6ca" containerName="revoke-keys" Oct 30 00:56:15.887909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.887881 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c35ed98-93b1-4adc-8e86-565bce5af6ca" containerName="revoke-keys" Oct 30 00:56:15.887998 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.887979 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3a486f-8f39-4cf0-8113-aacea94f695b" containerName="revoke-keys" Oct 30 00:56:15.887998 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.887988 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c35ed98-93b1-4adc-8e86-565bce5af6ca" containerName="revoke-keys" Oct 30 00:56:15.891688 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.891667 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:15.912816 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.912793 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4"] Oct 30 00:56:15.919378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.919256 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:56:15.919378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.919237 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:56:15.919378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.919347 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:56:15.921066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.921047 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:56:15.955685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.955657 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:15.955796 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.955709 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm26\" (UniqueName: \"kubernetes.io/projected/d045905c-65eb-4e9b-92e8-121efb05f49f-kube-api-access-zjm26\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:15.955852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:15.955832 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.056684 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.056658 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.056832 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.056701 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm26\" (UniqueName: \"kubernetes.io/projected/d045905c-65eb-4e9b-92e8-121efb05f49f-kube-api-access-zjm26\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.056832 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.056787 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.057369 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.057344 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.059329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.059306 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.078879 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.078858 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm26\" (UniqueName: \"kubernetes.io/projected/d045905c-65eb-4e9b-92e8-121efb05f49f-kube-api-access-zjm26\") pod \"maas-api-revoke-sub-037a9438-jdsl4\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.206594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.205740 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:16.571436 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:16.571411 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd045905c_65eb_4e9b_92e8_121efb05f49f.slice/crio-671388d150483470b20b03108aad374badadd3a8a44b7f47c0c5ddace0d72210 WatchSource:0}: Error finding container 671388d150483470b20b03108aad374badadd3a8a44b7f47c0c5ddace0d72210: Status 404 returned error can't find the container with id 671388d150483470b20b03108aad374badadd3a8a44b7f47c0c5ddace0d72210 Oct 30 00:56:16.574917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.574893 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4"] Oct 30 00:56:16.741442 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:16.741413 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" event={"ID":"d045905c-65eb-4e9b-92e8-121efb05f49f","Type":"ContainerStarted","Data":"671388d150483470b20b03108aad374badadd3a8a44b7f47c0c5ddace0d72210"} Oct 30 00:56:17.750989 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:17.750955 4905 generic.go:356] "Generic (PLEG): container finished" podID="d045905c-65eb-4e9b-92e8-121efb05f49f" containerID="e9f8c8d6bac77313686ad9bb8787b9cb28f395e2751633ab92fb97014e2a2878" exitCode=0 Oct 30 00:56:17.751371 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:17.751007 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" event={"ID":"d045905c-65eb-4e9b-92e8-121efb05f49f","Type":"ContainerDied","Data":"e9f8c8d6bac77313686ad9bb8787b9cb28f395e2751633ab92fb97014e2a2878"} Oct 30 00:56:18.690164 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.690129 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw"] Oct 30 00:56:18.694457 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.694436 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.715722 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.715694 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw"] Oct 30 00:56:18.798932 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.798901 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.799313 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.799028 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwgv\" (UniqueName: \"kubernetes.io/projected/fcc21080-9ea1-4af3-b050-deba98eeec85-kube-api-access-jvwgv\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.799313 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.799120 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.900678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.900645 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.900855 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.900767 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwgv\" (UniqueName: \"kubernetes.io/projected/fcc21080-9ea1-4af3-b050-deba98eeec85-kube-api-access-jvwgv\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.900938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.900857 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.901390 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.901363 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.903324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.903299 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.920959 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.920938 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwgv\" (UniqueName: \"kubernetes.io/projected/fcc21080-9ea1-4af3-b050-deba98eeec85-kube-api-access-jvwgv\") pod \"maas-api-revoke-sub-00bc1f61-vpgnw\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:18.944058 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:18.944002 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:19.004673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.004649 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-xks-ca\") pod \"d045905c-65eb-4e9b-92e8-121efb05f49f\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " Oct 30 00:56:19.004836 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.004820 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d045905c-65eb-4e9b-92e8-121efb05f49f-kube-api-access-zjm26\" (UniqueName: \"kubernetes.io/projected/d045905c-65eb-4e9b-92e8-121efb05f49f-kube-api-access-zjm26\") pod \"d045905c-65eb-4e9b-92e8-121efb05f49f\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " Oct 30 00:56:19.005066 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.004993 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-openshift-ca\") pod \"d045905c-65eb-4e9b-92e8-121efb05f49f\" (UID: \"d045905c-65eb-4e9b-92e8-121efb05f49f\") " Oct 30 00:56:19.005422 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.005392 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-openshift-ca" pod "d045905c-65eb-4e9b-92e8-121efb05f49f" (UID: "d045905c-65eb-4e9b-92e8-121efb05f49f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:19.008112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.007796 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-xks-ca" pod "d045905c-65eb-4e9b-92e8-121efb05f49f" (UID: "d045905c-65eb-4e9b-92e8-121efb05f49f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:19.008112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.007933 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:19.027206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.024917 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d045905c-65eb-4e9b-92e8-121efb05f49f-kube-api-access-zjm26" pod "d045905c-65eb-4e9b-92e8-121efb05f49f" (UID: "d045905c-65eb-4e9b-92e8-121efb05f49f"). InnerVolumeSpecName "kube-api-access-zjm26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:19.106917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.106891 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:19.107011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.106923 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045905c-65eb-4e9b-92e8-121efb05f49f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:19.107011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.106937 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm26\" (UniqueName: \"kubernetes.io/projected/d045905c-65eb-4e9b-92e8-121efb05f49f-kube-api-access-zjm26\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:19.334587 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:19.334558 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc21080_9ea1_4af3_b050_deba98eeec85.slice/crio-ae388ab0bfee63714f2289b80a0c03a270986fe59d671d1042190400866c577e WatchSource:0}: Error finding container ae388ab0bfee63714f2289b80a0c03a270986fe59d671d1042190400866c577e: Status 404 returned error can't find the container with id ae388ab0bfee63714f2289b80a0c03a270986fe59d671d1042190400866c577e Oct 30 00:56:19.336547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.336488 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw"] Oct 30 00:56:19.473798 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.473727 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7"] Oct 30 00:56:19.474404 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.474375 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d045905c-65eb-4e9b-92e8-121efb05f49f" containerName="revoke-keys" Oct 30 00:56:19.474510 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.474405 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d045905c-65eb-4e9b-92e8-121efb05f49f" containerName="revoke-keys" Oct 30 00:56:19.474679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.474662 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d045905c-65eb-4e9b-92e8-121efb05f49f" containerName="revoke-keys" Oct 30 00:56:19.479701 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.479682 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.497495 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.497470 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7"] Oct 30 00:56:19.619543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.616646 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.619543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.616846 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np9sw\" (UniqueName: \"kubernetes.io/projected/d806486c-d2b5-4e50-9610-4eba7e20d43d-kube-api-access-np9sw\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.619543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.616972 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.718811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.718774 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.719025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.718879 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np9sw\" (UniqueName: \"kubernetes.io/projected/d806486c-d2b5-4e50-9610-4eba7e20d43d-kube-api-access-np9sw\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.719025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.718943 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.719671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.719593 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.721357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.721332 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.739296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.739276 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np9sw\" (UniqueName: \"kubernetes.io/projected/d806486c-d2b5-4e50-9610-4eba7e20d43d-kube-api-access-np9sw\") pod \"maas-api-revoke-sub-87470f9e-mxls7\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.771086 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.771069 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" Oct 30 00:56:19.771193 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.771071 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4" event={"ID":"d045905c-65eb-4e9b-92e8-121efb05f49f","Type":"ContainerDied","Data":"671388d150483470b20b03108aad374badadd3a8a44b7f47c0c5ddace0d72210"} Oct 30 00:56:19.771251 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.771217 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671388d150483470b20b03108aad374badadd3a8a44b7f47c0c5ddace0d72210" Oct 30 00:56:19.772982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.772960 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" event={"ID":"fcc21080-9ea1-4af3-b050-deba98eeec85","Type":"ContainerStarted","Data":"48c3e2a3cd06f631f58c7b716c38110651795b18338115ef61c3b024379f57d7"} Oct 30 00:56:19.773067 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.773000 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" event={"ID":"fcc21080-9ea1-4af3-b050-deba98eeec85","Type":"ContainerStarted","Data":"ae388ab0bfee63714f2289b80a0c03a270986fe59d671d1042190400866c577e"} Oct 30 00:56:19.793874 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.793848 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:19.819665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:19.819618 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" podStartSLOduration=1.819603045 podStartE2EDuration="1.819603045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:19.812802774 +0000 UTC m=+1064.755975965" watchObservedRunningTime="2026-10-30 00:56:19.819603045 +0000 UTC m=+1064.762776237" Oct 30 00:56:20.096666 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:20.096623 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd806486c_d2b5_4e50_9610_4eba7e20d43d.slice/crio-84c918c7780c2b280c7140d2046ff0bffc96999c5e09406b6df15c8f89400112 WatchSource:0}: Error finding container 84c918c7780c2b280c7140d2046ff0bffc96999c5e09406b6df15c8f89400112: Status 404 returned error can't find the container with id 84c918c7780c2b280c7140d2046ff0bffc96999c5e09406b6df15c8f89400112 Oct 30 00:56:20.109548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:20.109505 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7"] Oct 30 00:56:20.783477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:20.783446 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" event={"ID":"d806486c-d2b5-4e50-9610-4eba7e20d43d","Type":"ContainerStarted","Data":"44ad240404b4a84ae272f5b381417a4f812083189a858b4a2d1775e770950ffa"} Oct 30 00:56:20.783657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:20.783510 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" event={"ID":"d806486c-d2b5-4e50-9610-4eba7e20d43d","Type":"ContainerStarted","Data":"84c918c7780c2b280c7140d2046ff0bffc96999c5e09406b6df15c8f89400112"} Oct 30 00:56:20.785163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:20.785136 4905 generic.go:356] "Generic (PLEG): container finished" podID="fcc21080-9ea1-4af3-b050-deba98eeec85" containerID="48c3e2a3cd06f631f58c7b716c38110651795b18338115ef61c3b024379f57d7" exitCode=0 Oct 30 00:56:20.785304 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:20.785221 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" event={"ID":"fcc21080-9ea1-4af3-b050-deba98eeec85","Type":"ContainerDied","Data":"48c3e2a3cd06f631f58c7b716c38110651795b18338115ef61c3b024379f57d7"} Oct 30 00:56:20.830165 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:20.830110 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" podStartSLOduration=1.83009271 podStartE2EDuration="1.83009271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:20.823434983 +0000 UTC m=+1065.766608174" watchObservedRunningTime="2026-10-30 00:56:20.83009271 +0000 UTC m=+1065.773265903" Oct 30 00:56:21.794475 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:21.794446 4905 generic.go:356] "Generic (PLEG): container finished" podID="d806486c-d2b5-4e50-9610-4eba7e20d43d" containerID="44ad240404b4a84ae272f5b381417a4f812083189a858b4a2d1775e770950ffa" exitCode=0 Oct 30 00:56:21.794677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:21.794543 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" event={"ID":"d806486c-d2b5-4e50-9610-4eba7e20d43d","Type":"ContainerDied","Data":"44ad240404b4a84ae272f5b381417a4f812083189a858b4a2d1775e770950ffa"} Oct 30 00:56:22.005579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.005553 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:22.157426 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.157343 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-openshift-ca\") pod \"fcc21080-9ea1-4af3-b050-deba98eeec85\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " Oct 30 00:56:22.157606 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.157447 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc21080-9ea1-4af3-b050-deba98eeec85-kube-api-access-jvwgv\" (UniqueName: \"kubernetes.io/projected/fcc21080-9ea1-4af3-b050-deba98eeec85-kube-api-access-jvwgv\") pod \"fcc21080-9ea1-4af3-b050-deba98eeec85\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " Oct 30 00:56:22.157687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.157639 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-xks-ca\") pod \"fcc21080-9ea1-4af3-b050-deba98eeec85\" (UID: \"fcc21080-9ea1-4af3-b050-deba98eeec85\") " Oct 30 00:56:22.157779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.157750 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-openshift-ca" pod "fcc21080-9ea1-4af3-b050-deba98eeec85" (UID: "fcc21080-9ea1-4af3-b050-deba98eeec85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:22.158315 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.158290 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:22.160390 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.160361 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-xks-ca" pod "fcc21080-9ea1-4af3-b050-deba98eeec85" (UID: "fcc21080-9ea1-4af3-b050-deba98eeec85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:22.160471 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.160411 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc21080-9ea1-4af3-b050-deba98eeec85-kube-api-access-jvwgv" pod "fcc21080-9ea1-4af3-b050-deba98eeec85" (UID: "fcc21080-9ea1-4af3-b050-deba98eeec85"). InnerVolumeSpecName "kube-api-access-jvwgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:22.260169 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.260139 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvwgv\" (UniqueName: \"kubernetes.io/projected/fcc21080-9ea1-4af3-b050-deba98eeec85-kube-api-access-jvwgv\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:22.260304 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.260172 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc21080-9ea1-4af3-b050-deba98eeec85-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:22.803634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.803605 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" Oct 30 00:56:22.803799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.803636 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw" event={"ID":"fcc21080-9ea1-4af3-b050-deba98eeec85","Type":"ContainerDied","Data":"ae388ab0bfee63714f2289b80a0c03a270986fe59d671d1042190400866c577e"} Oct 30 00:56:22.803799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.803683 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae388ab0bfee63714f2289b80a0c03a270986fe59d671d1042190400866c577e" Oct 30 00:56:22.984352 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:22.984325 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:23.075869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.075782 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-openshift-ca\") pod \"d806486c-d2b5-4e50-9610-4eba7e20d43d\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " Oct 30 00:56:23.076275 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.075918 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-xks-ca\") pod \"d806486c-d2b5-4e50-9610-4eba7e20d43d\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " Oct 30 00:56:23.076275 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.075950 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d806486c-d2b5-4e50-9610-4eba7e20d43d-kube-api-access-np9sw\" (UniqueName: \"kubernetes.io/projected/d806486c-d2b5-4e50-9610-4eba7e20d43d-kube-api-access-np9sw\") pod \"d806486c-d2b5-4e50-9610-4eba7e20d43d\" (UID: \"d806486c-d2b5-4e50-9610-4eba7e20d43d\") " Oct 30 00:56:23.076879 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.076848 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-openshift-ca" pod "d806486c-d2b5-4e50-9610-4eba7e20d43d" (UID: "d806486c-d2b5-4e50-9610-4eba7e20d43d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:23.078854 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.078813 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d806486c-d2b5-4e50-9610-4eba7e20d43d-kube-api-access-np9sw" pod "d806486c-d2b5-4e50-9610-4eba7e20d43d" (UID: "d806486c-d2b5-4e50-9610-4eba7e20d43d"). InnerVolumeSpecName "kube-api-access-np9sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:23.085658 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.085635 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-xks-ca" pod "d806486c-d2b5-4e50-9610-4eba7e20d43d" (UID: "d806486c-d2b5-4e50-9610-4eba7e20d43d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:23.177993 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.177969 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:23.177993 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.177992 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d806486c-d2b5-4e50-9610-4eba7e20d43d-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:23.178135 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.178002 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np9sw\" (UniqueName: \"kubernetes.io/projected/d806486c-d2b5-4e50-9610-4eba7e20d43d-kube-api-access-np9sw\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:23.812269 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.812236 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" Oct 30 00:56:23.812441 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.812272 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7" event={"ID":"d806486c-d2b5-4e50-9610-4eba7e20d43d","Type":"ContainerDied","Data":"84c918c7780c2b280c7140d2046ff0bffc96999c5e09406b6df15c8f89400112"} Oct 30 00:56:23.812441 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:23.812317 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84c918c7780c2b280c7140d2046ff0bffc96999c5e09406b6df15c8f89400112" Oct 30 00:56:26.258545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.249573 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8"] Oct 30 00:56:26.258545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.250504 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d806486c-d2b5-4e50-9610-4eba7e20d43d" containerName="revoke-keys" Oct 30 00:56:26.258545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.250545 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d806486c-d2b5-4e50-9610-4eba7e20d43d" containerName="revoke-keys" Oct 30 00:56:26.258545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.250567 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fcc21080-9ea1-4af3-b050-deba98eeec85" containerName="revoke-keys" Oct 30 00:56:26.258545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.250575 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc21080-9ea1-4af3-b050-deba98eeec85" containerName="revoke-keys" Oct 30 00:56:26.258545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.250791 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d806486c-d2b5-4e50-9610-4eba7e20d43d" containerName="revoke-keys" Oct 30 00:56:26.258545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.250804 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc21080-9ea1-4af3-b050-deba98eeec85" containerName="revoke-keys" Oct 30 00:56:26.259982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.259920 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.270055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.263067 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8"] Oct 30 00:56:26.299543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.298824 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:56:26.299543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.299487 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:56:26.299787 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.299604 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:56:26.299787 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.299666 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:56:26.403592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.363082 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscrm\" (UniqueName: \"kubernetes.io/projected/75e04f1f-b10b-49cc-a71b-77c7fee573d0-kube-api-access-hscrm\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.417561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.403964 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.417885 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.417856 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.535593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.519810 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscrm\" (UniqueName: \"kubernetes.io/projected/75e04f1f-b10b-49cc-a71b-77c7fee573d0-kube-api-access-hscrm\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.615732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.615692 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.616039 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.616017 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.662541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.633460 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.720625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.720474 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscrm\" (UniqueName: \"kubernetes.io/projected/75e04f1f-b10b-49cc-a71b-77c7fee573d0-kube-api-access-hscrm\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.728027 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.727993 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328f54ae-wdft8\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:26.965862 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:26.965776 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:27.972546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:27.935974 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:56:28.159554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.016635 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:28.159554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.029130 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:56:28.246657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.171176 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:56:28.246657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.231603 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:28.566578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.566538 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:28.601549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.597229 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:28.602858 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:28.602830 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e04f1f_b10b_49cc_a71b_77c7fee573d0.slice/crio-d8e3b9b3c22564942266a145ea6100675c399b8780e054a2da26a7b9600f8426 WatchSource:0}: Error finding container d8e3b9b3c22564942266a145ea6100675c399b8780e054a2da26a7b9600f8426: Status 404 returned error can't find the container with id d8e3b9b3c22564942266a145ea6100675c399b8780e054a2da26a7b9600f8426 Oct 30 00:56:28.609542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.603372 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8"] Oct 30 00:56:28.982357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:28.982299 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:29.027824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.027780 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:29.029532 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.029480 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:29.100652 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.100597 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:29.101480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.101459 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:56:29.245729 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.245629 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:56:29.246276 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.246243 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:56:29.247049 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.247030 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:56:29.270950 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.248164 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:56:29.271242 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.267631 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:29.646062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.610248 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:29.646062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.632851 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:56:29.646062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.632905 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:56:29.646062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.632946 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:56:29.646062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.632979 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:56:29.646062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.633845 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:29.646062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:29.634158 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:30.106687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.106642 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:30.207581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.192132 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:30.300800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.300759 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:30.422352 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.422069 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:30.423897 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.423868 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:56:30.424331 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.424306 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:56:30.459577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.459469 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:56:30.594535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.594479 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:56:30.594843 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.594823 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:56:30.681594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.671846 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:30.809544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.801316 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" podUID="34ba07ff-2d49-4483-a644-04ce768e4937" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:30.809544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:30.801456 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:56:31.036546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.023062 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:31.194656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.194607 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:31.347602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.346434 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:31.464202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.432120 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:31.464202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.432512 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:56:31.464202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.432781 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 30 00:56:31.464202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.432827 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:56:31.464202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.448733 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:56:31.464202 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.448885 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" event={"ID":"75e04f1f-b10b-49cc-a71b-77c7fee573d0","Type":"ContainerStarted","Data":"d8e3b9b3c22564942266a145ea6100675c399b8780e054a2da26a7b9600f8426"} Oct 30 00:56:31.490737 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.490670 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:31.795129 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.687841 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:56:31.821328 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.821275 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:31.828932 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.828897 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:31.849332 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.849304 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx"] Oct 30 00:56:31.856966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.856942 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx"] Oct 30 00:56:31.857221 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.857207 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.928335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.928299 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.929108 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.929080 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d523e-4a52-4960-9564-912a704258d2-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.929282 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.929268 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.929394 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.929379 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.929502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.929477 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.929653 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.929637 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.929824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.929809 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.929959 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.929945 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvrss\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-kube-api-access-cvrss\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.931562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.931542 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d523e-4a52-4960-9564-912a704258d2-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:31.994772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:31.994731 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:32.023396 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.023357 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:56:32.089594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.088658 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvrss\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-kube-api-access-cvrss\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.089594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.088897 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d523e-4a52-4960-9564-912a704258d2-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.107590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.097351 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.107590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.105035 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.116538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.112592 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d523e-4a52-4960-9564-912a704258d2-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.116817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.116617 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.143013 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.142976 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.179641 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.167090 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.198665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.198614 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.199278 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.199256 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.199500 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.199484 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.200137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.200108 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.217574 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.206995 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d523e-4a52-4960-9564-912a704258d2-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.217932 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.217884 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.275546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.253859 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.275546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.254288 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvrss\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-kube-api-access-cvrss\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.275546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.274737 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:56:32.353883 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.353761 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:32.374872 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.374828 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d523e-4a52-4960-9564-912a704258d2-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.376156 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.376134 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:56:32.379842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.379825 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:56:32.394067 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.394042 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:56:32.529540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.505983 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:32.529540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.506285 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:32.534537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.533296 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:56:32.534537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.533733 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:32.604115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.604034 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:32.608327 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.608287 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:56:32.608549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.608351 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:32.624770 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.624745 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:56:32.665986 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.665960 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:56:32.723621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.719785 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:56:32.723621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.720682 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:56:32.740459 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.730544 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:56:32.740459 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.732751 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:32.760546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.760243 4905 generic.go:356] "Generic (PLEG): container finished" podID="75e04f1f-b10b-49cc-a71b-77c7fee573d0" containerID="4766157e0bbbebf8866f78695525bdd057d86cadefe621b0225350f57d3122df" exitCode=0 Oct 30 00:56:32.773621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.772005 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" event={"ID":"75e04f1f-b10b-49cc-a71b-77c7fee573d0","Type":"ContainerDied","Data":"4766157e0bbbebf8866f78695525bdd057d86cadefe621b0225350f57d3122df"} Oct 30 00:56:32.826729 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:32.826696 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:33.184618 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:33.184576 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804d523e_4a52_4960_9564_912a704258d2.slice/crio-2f424e2fc92d1fdeb1e6125b8058c93ecf0cec4c607888bf773220066ce6ed4e WatchSource:0}: Error finding container 2f424e2fc92d1fdeb1e6125b8058c93ecf0cec4c607888bf773220066ce6ed4e: Status 404 returned error can't find the container with id 2f424e2fc92d1fdeb1e6125b8058c93ecf0cec4c607888bf773220066ce6ed4e Oct 30 00:56:33.189383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.189350 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:33.189489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.189413 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:33.189489 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.189441 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:33.196125 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.196088 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx"] Oct 30 00:56:33.771996 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.771952 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" event={"ID":"804d523e-4a52-4960-9564-912a704258d2","Type":"ContainerStarted","Data":"e4e39fef92c046edfd69bf920a93b24e55e092a3eebebaa7685a1ea90b304df1"} Oct 30 00:56:33.772466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.772019 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" event={"ID":"804d523e-4a52-4960-9564-912a704258d2","Type":"ContainerStarted","Data":"2f424e2fc92d1fdeb1e6125b8058c93ecf0cec4c607888bf773220066ce6ed4e"} Oct 30 00:56:33.821178 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.821111 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podStartSLOduration=3.8210906529999997 podStartE2EDuration="3.821090653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:33.814933321 +0000 UTC m=+1078.758106540" watchObservedRunningTime="2026-10-30 00:56:33.821090653 +0000 UTC m=+1078.764263847" Oct 30 00:56:33.827157 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.827127 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:33.996039 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:33.996017 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:34.068051 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.067964 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-xks-ca\") pod \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " Oct 30 00:56:34.068219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.068054 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-openshift-ca\") pod \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " Oct 30 00:56:34.068219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.068107 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e04f1f-b10b-49cc-a71b-77c7fee573d0-kube-api-access-hscrm\" (UniqueName: \"kubernetes.io/projected/75e04f1f-b10b-49cc-a71b-77c7fee573d0-kube-api-access-hscrm\") pod \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\" (UID: \"75e04f1f-b10b-49cc-a71b-77c7fee573d0\") " Oct 30 00:56:34.069556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.069397 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-openshift-ca" pod "75e04f1f-b10b-49cc-a71b-77c7fee573d0" (UID: "75e04f1f-b10b-49cc-a71b-77c7fee573d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:34.075810 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.075772 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e04f1f-b10b-49cc-a71b-77c7fee573d0-kube-api-access-hscrm" pod "75e04f1f-b10b-49cc-a71b-77c7fee573d0" (UID: "75e04f1f-b10b-49cc-a71b-77c7fee573d0"). InnerVolumeSpecName "kube-api-access-hscrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:34.082015 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.081795 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-xks-ca" pod "75e04f1f-b10b-49cc-a71b-77c7fee573d0" (UID: "75e04f1f-b10b-49cc-a71b-77c7fee573d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:34.107713 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.107680 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh"] Oct 30 00:56:34.108602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.108582 4905 cpu_manager.go:414] "containerMap: removing container" podUID="75e04f1f-b10b-49cc-a71b-77c7fee573d0" containerName="revoke-keys" Oct 30 00:56:34.108602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.108602 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e04f1f-b10b-49cc-a71b-77c7fee573d0" containerName="revoke-keys" Oct 30 00:56:34.108803 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.108792 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e04f1f-b10b-49cc-a71b-77c7fee573d0" containerName="revoke-keys" Oct 30 00:56:34.113829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.113809 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.173552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.170739 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:34.173552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.170773 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e04f1f-b10b-49cc-a71b-77c7fee573d0-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:34.173552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.170788 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hscrm\" (UniqueName: \"kubernetes.io/projected/75e04f1f-b10b-49cc-a71b-77c7fee573d0-kube-api-access-hscrm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:34.175380 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.175296 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh"] Oct 30 00:56:34.272134 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.272098 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.272448 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.272423 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.272590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.272478 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmdl\" (UniqueName: \"kubernetes.io/projected/c63b0a10-feb0-42c0-87b6-33bece8d721d-kube-api-access-2mmdl\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.374353 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.374276 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.374498 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.374447 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.374498 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.374476 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmdl\" (UniqueName: \"kubernetes.io/projected/c63b0a10-feb0-42c0-87b6-33bece8d721d-kube-api-access-2mmdl\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.375214 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.375191 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.376861 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.376837 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.394791 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.394765 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmdl\" (UniqueName: \"kubernetes.io/projected/c63b0a10-feb0-42c0-87b6-33bece8d721d-kube-api-access-2mmdl\") pod \"maas-api-revoke-sub-98984103-9j2mh\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.480975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.480943 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:34.785493 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.785466 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" Oct 30 00:56:34.785894 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.785499 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8" event={"ID":"75e04f1f-b10b-49cc-a71b-77c7fee573d0","Type":"ContainerDied","Data":"d8e3b9b3c22564942266a145ea6100675c399b8780e054a2da26a7b9600f8426"} Oct 30 00:56:34.785894 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.785567 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8e3b9b3c22564942266a145ea6100675c399b8780e054a2da26a7b9600f8426" Oct 30 00:56:34.816719 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:34.816690 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63b0a10_feb0_42c0_87b6_33bece8d721d.slice/crio-bef7a432df2ae06d50e5ba79c612bee7873da73d09d73358719e5bf99e9a3cdf WatchSource:0}: Error finding container bef7a432df2ae06d50e5ba79c612bee7873da73d09d73358719e5bf99e9a3cdf: Status 404 returned error can't find the container with id bef7a432df2ae06d50e5ba79c612bee7873da73d09d73358719e5bf99e9a3cdf Oct 30 00:56:34.820162 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.820142 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh"] Oct 30 00:56:34.868728 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.868685 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 00:56:34.868905 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:34.868765 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 00:56:35.174064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.174024 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr"] Oct 30 00:56:35.180481 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.180457 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.200077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.199507 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr"] Oct 30 00:56:35.360671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.302398 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzlx\" (UniqueName: \"kubernetes.io/projected/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-kube-api-access-klzlx\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.360671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.302462 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.360671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.302508 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.438792 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.438741 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzlx\" (UniqueName: \"kubernetes.io/projected/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-kube-api-access-klzlx\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.438946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.438818 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.438946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.438874 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.456601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.456551 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.460006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.459980 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.471566 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.469760 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzlx\" (UniqueName: \"kubernetes.io/projected/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-kube-api-access-klzlx\") pod \"maas-api-revoke-sub-29d94875-dg5lr\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.498402 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.498371 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:35.795987 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.795949 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" event={"ID":"c63b0a10-feb0-42c0-87b6-33bece8d721d","Type":"ContainerStarted","Data":"3b303c46d8e28ae206e0e6ecf512c0d03fe11fe4087de38ca4d242d7dd9d4506"} Oct 30 00:56:35.796425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.796046 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" event={"ID":"c63b0a10-feb0-42c0-87b6-33bece8d721d","Type":"ContainerStarted","Data":"bef7a432df2ae06d50e5ba79c612bee7873da73d09d73358719e5bf99e9a3cdf"} Oct 30 00:56:35.857720 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.857656 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 00:56:35.857799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.857708 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 00:56:35.859640 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.859613 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr"] Oct 30 00:56:35.861104 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:35.861079 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5048b5e_7d50_4aeb_9de1_d0f2d8a8f4b5.slice/crio-9c432505442a9c80b7baf0074b1ba9ea42da8d807a297e2fb2ae1eee15d69e26 WatchSource:0}: Error finding container 9c432505442a9c80b7baf0074b1ba9ea42da8d807a297e2fb2ae1eee15d69e26: Status 404 returned error can't find the container with id 9c432505442a9c80b7baf0074b1ba9ea42da8d807a297e2fb2ae1eee15d69e26 Oct 30 00:56:35.863493 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:35.863449 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" podStartSLOduration=1.863433136 podStartE2EDuration="1.863433136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:35.858073953 +0000 UTC m=+1080.801247144" watchObservedRunningTime="2026-10-30 00:56:35.863433136 +0000 UTC m=+1080.806606328" Oct 30 00:56:36.809450 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:36.809396 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" event={"ID":"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5","Type":"ContainerStarted","Data":"81a2f128f84492863eb695e489183b41f586497b592c08fb9b1bac6a5aff29b2"} Oct 30 00:56:36.809941 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:36.809470 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" event={"ID":"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5","Type":"ContainerStarted","Data":"9c432505442a9c80b7baf0074b1ba9ea42da8d807a297e2fb2ae1eee15d69e26"} Oct 30 00:56:36.811282 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:36.811250 4905 generic.go:356] "Generic (PLEG): container finished" podID="c63b0a10-feb0-42c0-87b6-33bece8d721d" containerID="3b303c46d8e28ae206e0e6ecf512c0d03fe11fe4087de38ca4d242d7dd9d4506" exitCode=0 Oct 30 00:56:36.811398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:36.811322 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" event={"ID":"c63b0a10-feb0-42c0-87b6-33bece8d721d","Type":"ContainerDied","Data":"3b303c46d8e28ae206e0e6ecf512c0d03fe11fe4087de38ca4d242d7dd9d4506"} Oct 30 00:56:36.842541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:36.840705 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 00:56:36.842541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:36.840757 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 00:56:36.866306 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:36.866263 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" podStartSLOduration=1.866246998 podStartE2EDuration="1.866246998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:36.865793156 +0000 UTC m=+1081.808966347" watchObservedRunningTime="2026-10-30 00:56:36.866246998 +0000 UTC m=+1081.809420190" Oct 30 00:56:37.820747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:37.820711 4905 generic.go:356] "Generic (PLEG): container finished" podID="b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" containerID="81a2f128f84492863eb695e489183b41f586497b592c08fb9b1bac6a5aff29b2" exitCode=0 Oct 30 00:56:37.821135 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:37.820800 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" event={"ID":"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5","Type":"ContainerDied","Data":"81a2f128f84492863eb695e489183b41f586497b592c08fb9b1bac6a5aff29b2"} Oct 30 00:56:37.835543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:37.834840 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 00:56:37.835543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:37.834895 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 00:56:38.083940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.083868 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:38.210731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.210699 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-xks-ca\") pod \"c63b0a10-feb0-42c0-87b6-33bece8d721d\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " Oct 30 00:56:38.210960 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.210836 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-openshift-ca\") pod \"c63b0a10-feb0-42c0-87b6-33bece8d721d\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " Oct 30 00:56:38.211033 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.210962 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63b0a10-feb0-42c0-87b6-33bece8d721d-kube-api-access-2mmdl\" (UniqueName: \"kubernetes.io/projected/c63b0a10-feb0-42c0-87b6-33bece8d721d-kube-api-access-2mmdl\") pod \"c63b0a10-feb0-42c0-87b6-33bece8d721d\" (UID: \"c63b0a10-feb0-42c0-87b6-33bece8d721d\") " Oct 30 00:56:38.211315 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.211286 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-openshift-ca" pod "c63b0a10-feb0-42c0-87b6-33bece8d721d" (UID: "c63b0a10-feb0-42c0-87b6-33bece8d721d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:38.211626 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.211606 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:38.214546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.213504 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-xks-ca" pod "c63b0a10-feb0-42c0-87b6-33bece8d721d" (UID: "c63b0a10-feb0-42c0-87b6-33bece8d721d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:38.214546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.213652 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63b0a10-feb0-42c0-87b6-33bece8d721d-kube-api-access-2mmdl" pod "c63b0a10-feb0-42c0-87b6-33bece8d721d" (UID: "c63b0a10-feb0-42c0-87b6-33bece8d721d"). InnerVolumeSpecName "kube-api-access-2mmdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:38.313300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.313251 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mmdl\" (UniqueName: \"kubernetes.io/projected/c63b0a10-feb0-42c0-87b6-33bece8d721d-kube-api-access-2mmdl\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:38.313598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.313306 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b0a10-feb0-42c0-87b6-33bece8d721d-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:38.408918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.408848 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl"] Oct 30 00:56:38.409571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.409548 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c63b0a10-feb0-42c0-87b6-33bece8d721d" containerName="revoke-keys" Oct 30 00:56:38.409571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.409565 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63b0a10-feb0-42c0-87b6-33bece8d721d" containerName="revoke-keys" Oct 30 00:56:38.409713 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.409696 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63b0a10-feb0-42c0-87b6-33bece8d721d" containerName="revoke-keys" Oct 30 00:56:38.413584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.413557 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.481781 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.481754 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl"] Oct 30 00:56:38.516337 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.516302 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ccr\" (UniqueName: \"kubernetes.io/projected/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-kube-api-access-c4ccr\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.516509 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.516489 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.516752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.516735 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.618608 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.618571 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.618772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.618659 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ccr\" (UniqueName: \"kubernetes.io/projected/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-kube-api-access-c4ccr\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.618821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.618770 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.619367 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.619343 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.621147 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.621125 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.656545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.656131 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ccr\" (UniqueName: \"kubernetes.io/projected/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-kube-api-access-c4ccr\") pod \"maas-api-revoke-sub-9349751a-w22gl\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.745593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.743208 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:38.837541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.830672 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 00:56:38.837541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.830724 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 00:56:38.860944 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.860914 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" Oct 30 00:56:38.868912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.868813 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh" event={"ID":"c63b0a10-feb0-42c0-87b6-33bece8d721d","Type":"ContainerDied","Data":"bef7a432df2ae06d50e5ba79c612bee7873da73d09d73358719e5bf99e9a3cdf"} Oct 30 00:56:38.869086 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:38.869069 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef7a432df2ae06d50e5ba79c612bee7873da73d09d73358719e5bf99e9a3cdf" Oct 30 00:56:39.191282 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:39.191184 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b12fda0_42b7_49a9_86db_97fb4d3a2b0e.slice/crio-8f6b4ca7aa568b8ce6c330390694e517b15266c19ec625ad86af2d9514e51400 WatchSource:0}: Error finding container 8f6b4ca7aa568b8ce6c330390694e517b15266c19ec625ad86af2d9514e51400: Status 404 returned error can't find the container with id 8f6b4ca7aa568b8ce6c330390694e517b15266c19ec625ad86af2d9514e51400 Oct 30 00:56:39.215975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.215955 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:39.216710 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.216690 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl"] Oct 30 00:56:39.367621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.367530 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-kube-api-access-klzlx\" (UniqueName: \"kubernetes.io/projected/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-kube-api-access-klzlx\") pod \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " Oct 30 00:56:39.367621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.367590 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-openshift-ca\") pod \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " Oct 30 00:56:39.367864 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.367639 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-xks-ca\") pod \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\" (UID: \"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5\") " Oct 30 00:56:39.368020 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.367987 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-openshift-ca" pod "b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" (UID: "b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:39.368332 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.368309 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:39.426215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.426186 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-kube-api-access-klzlx" pod "b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" (UID: "b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5"). InnerVolumeSpecName "kube-api-access-klzlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:39.426412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.426215 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-xks-ca" pod "b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" (UID: "b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:39.469697 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.469667 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klzlx\" (UniqueName: \"kubernetes.io/projected/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-kube-api-access-klzlx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:39.469800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.469700 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:39.827294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.827257 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 00:56:39.827488 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.827314 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 00:56:39.871762 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.871736 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" Oct 30 00:56:39.872197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.871761 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr" event={"ID":"b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5","Type":"ContainerDied","Data":"9c432505442a9c80b7baf0074b1ba9ea42da8d807a297e2fb2ae1eee15d69e26"} Oct 30 00:56:39.872197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.871814 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c432505442a9c80b7baf0074b1ba9ea42da8d807a297e2fb2ae1eee15d69e26" Oct 30 00:56:39.873706 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.873683 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" event={"ID":"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e","Type":"ContainerStarted","Data":"019577037a79d8fc607f6ec9a2eee53514ae8650d36f025622862cdee310a325"} Oct 30 00:56:39.873828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.873728 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" event={"ID":"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e","Type":"ContainerStarted","Data":"8f6b4ca7aa568b8ce6c330390694e517b15266c19ec625ad86af2d9514e51400"} Oct 30 00:56:39.913951 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:39.913902 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" podStartSLOduration=1.913886395 podStartE2EDuration="1.913886395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:39.913122917 +0000 UTC m=+1084.856296105" watchObservedRunningTime="2026-10-30 00:56:39.913886395 +0000 UTC m=+1084.857059591" Oct 30 00:56:40.831949 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:40.831918 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:40.832412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:40.832396 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:40.833251 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:40.833229 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 00:56:40.883254 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:40.883221 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" containerID="019577037a79d8fc607f6ec9a2eee53514ae8650d36f025622862cdee310a325" exitCode=0 Oct 30 00:56:40.883614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:40.883268 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" event={"ID":"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e","Type":"ContainerDied","Data":"019577037a79d8fc607f6ec9a2eee53514ae8650d36f025622862cdee310a325"} Oct 30 00:56:42.167235 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.167208 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:42.220484 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.220449 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-openshift-ca\") pod \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " Oct 30 00:56:42.220642 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.220626 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-kube-api-access-c4ccr\" (UniqueName: \"kubernetes.io/projected/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-kube-api-access-c4ccr\") pod \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " Oct 30 00:56:42.220840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.220818 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-openshift-ca" pod "2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" (UID: "2b12fda0-42b7-49a9-86db-97fb4d3a2b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:42.220939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.220857 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-xks-ca\") pod \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\" (UID: \"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e\") " Oct 30 00:56:42.221472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.221450 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:42.222983 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.222961 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-xks-ca" pod "2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" (UID: "2b12fda0-42b7-49a9-86db-97fb4d3a2b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:42.223146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.223130 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-kube-api-access-c4ccr" pod "2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" (UID: "2b12fda0-42b7-49a9-86db-97fb4d3a2b0e"). InnerVolumeSpecName "kube-api-access-c4ccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:42.323418 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.323334 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:42.323418 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.323363 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4ccr\" (UniqueName: \"kubernetes.io/projected/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e-kube-api-access-c4ccr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:42.414364 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:56:42.414327 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:56:42.992220 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.992182 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" event={"ID":"2b12fda0-42b7-49a9-86db-97fb4d3a2b0e","Type":"ContainerDied","Data":"8f6b4ca7aa568b8ce6c330390694e517b15266c19ec625ad86af2d9514e51400"} Oct 30 00:56:42.992406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.992223 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl" Oct 30 00:56:42.992406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:42.992230 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f6b4ca7aa568b8ce6c330390694e517b15266c19ec625ad86af2d9514e51400" Oct 30 00:56:43.932458 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.932425 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk"] Oct 30 00:56:43.933064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.933047 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" containerName="revoke-keys" Oct 30 00:56:43.933064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.933063 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" containerName="revoke-keys" Oct 30 00:56:43.933183 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.933081 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" containerName="revoke-keys" Oct 30 00:56:43.933183 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.933086 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" containerName="revoke-keys" Oct 30 00:56:43.933291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.933201 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" containerName="revoke-keys" Oct 30 00:56:43.933291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.933217 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" containerName="revoke-keys" Oct 30 00:56:43.937006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.936986 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:43.957070 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.957046 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:56:43.957219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.957075 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:56:43.974562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.971120 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:56:43.974562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.971118 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:56:43.977277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:43.977255 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk"] Oct 30 00:56:44.048421 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.048395 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mrh\" (UniqueName: \"kubernetes.io/projected/66f77110-832f-4cc5-bff7-ae34fbfc719a-kube-api-access-s8mrh\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.048584 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.048560 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.048741 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.048725 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.150374 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.150341 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mrh\" (UniqueName: \"kubernetes.io/projected/66f77110-832f-4cc5-bff7-ae34fbfc719a-kube-api-access-s8mrh\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.150555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.150484 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.150656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.150639 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.151352 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.151318 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.153177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.153147 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.170700 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.170660 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mrh\" (UniqueName: \"kubernetes.io/projected/66f77110-832f-4cc5-bff7-ae34fbfc719a-kube-api-access-s8mrh\") pod \"maas-api-revoke-sub-c098b524-z57wk\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.250509 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.250483 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:44.553373 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:56:44.553343 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f77110_832f_4cc5_bff7_ae34fbfc719a.slice/crio-844a44d70783182b4f36a6d926c2e7e7bd9d6d6e1a66262774021c20e43398d8 WatchSource:0}: Error finding container 844a44d70783182b4f36a6d926c2e7e7bd9d6d6e1a66262774021c20e43398d8: Status 404 returned error can't find the container with id 844a44d70783182b4f36a6d926c2e7e7bd9d6d6e1a66262774021c20e43398d8 Oct 30 00:56:44.557103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:44.557083 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk"] Oct 30 00:56:45.010115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.010063 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" event={"ID":"66f77110-832f-4cc5-bff7-ae34fbfc719a","Type":"ContainerStarted","Data":"b44a9c50312d85cc8d19e13b0c0ea2c2b6f7d02e8f937b38c9df6dcd3e784f3a"} Oct 30 00:56:45.010556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.010127 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" event={"ID":"66f77110-832f-4cc5-bff7-ae34fbfc719a","Type":"ContainerStarted","Data":"844a44d70783182b4f36a6d926c2e7e7bd9d6d6e1a66262774021c20e43398d8"} Oct 30 00:56:45.067376 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.067332 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" podStartSLOduration=2.067318228 podStartE2EDuration="2.067318228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:45.051732821 +0000 UTC m=+1089.994906014" watchObservedRunningTime="2026-10-30 00:56:45.067318228 +0000 UTC m=+1090.010491420" Oct 30 00:56:45.871543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.871441 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn"] Oct 30 00:56:45.875533 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.875492 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.902295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.901675 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn"] Oct 30 00:56:45.975631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.975600 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4b60c4-d140-43ee-9c61-802cba188d6b-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.975796 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.975647 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.975796 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.975671 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.975875 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.975824 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jkr5\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-kube-api-access-4jkr5\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.975875 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.975856 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.976033 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.975986 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.976161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.976144 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.976197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.976186 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:45.976229 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:45.976213 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.019628 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.019596 4905 generic.go:356] "Generic (PLEG): container finished" podID="66f77110-832f-4cc5-bff7-ae34fbfc719a" containerID="b44a9c50312d85cc8d19e13b0c0ea2c2b6f7d02e8f937b38c9df6dcd3e784f3a" exitCode=0 Oct 30 00:56:46.020011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.019642 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" event={"ID":"66f77110-832f-4cc5-bff7-ae34fbfc719a","Type":"ContainerDied","Data":"b44a9c50312d85cc8d19e13b0c0ea2c2b6f7d02e8f937b38c9df6dcd3e784f3a"} Oct 30 00:56:46.078256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078181 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078236 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078484 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078265 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078611 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078583 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4b60c4-d140-43ee-9c61-802cba188d6b-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078651 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078682 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078691 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078861 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078824 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jkr5\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-kube-api-access-4jkr5\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078876 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.078974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078957 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.079013 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.078983 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.079353 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.079331 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.079415 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.079368 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4b60c4-d140-43ee-9c61-802cba188d6b-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.079769 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.079619 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.081528 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.081494 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.081721 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.081700 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.099147 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.099121 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.099259 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.099217 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jkr5\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-kube-api-access-4jkr5\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.192643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.192560 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:46.197445 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.197409 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2"] Oct 30 00:56:46.202101 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.202085 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.224821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.224792 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2"] Oct 30 00:56:46.242556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.235568 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc"] Oct 30 00:56:46.265799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.265654 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.277597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.277564 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc"] Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287375 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287537 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80abb66d-7e8c-4524-8902-e4afd8221305-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287583 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287658 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287694 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80abb66d-7e8c-4524-8902-e4afd8221305-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287768 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287804 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzssm\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-kube-api-access-bzssm\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287843 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.305702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.287875 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.388831 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80abb66d-7e8c-4524-8902-e4afd8221305-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.388895 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.388932 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.388997 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389034 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80abb66d-7e8c-4524-8902-e4afd8221305-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389061 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389217 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389312 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389336 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389352 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389436 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzssm\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-kube-api-access-bzssm\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389471 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389491 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389551 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389617 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvf4l\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-kube-api-access-vvf4l\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389777 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389807 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.389962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389841 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.389962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.389918 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.390297 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.390268 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80abb66d-7e8c-4524-8902-e4afd8221305-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.390430 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.390395 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.390499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.390446 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.390823 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.390767 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.392720 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.392698 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.395956 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.395934 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80abb66d-7e8c-4524-8902-e4afd8221305-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.414093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.414076 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.414876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.414843 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzssm\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-kube-api-access-bzssm\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.490811 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.490784 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.490953 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.490859 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.490953 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.490900 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvf4l\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-kube-api-access-vvf4l\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.490953 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.490921 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491075 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491041 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491153 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491269 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491253 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491348 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491326 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491388 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491355 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491384 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491407 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491555 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491588 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.491741 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.491718 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.493700 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.493678 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.493938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.493920 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.511153 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.511127 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvf4l\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-kube-api-access-vvf4l\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.511292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.511276 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.576078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.576050 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn"] Oct 30 00:56:46.576822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.576787 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:46.576904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.576853 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:46.576904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.576887 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:46.624004 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.623975 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:46.643638 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.643615 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:46.976679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.975976 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:46.976679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.976070 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:46.976679 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.976116 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:46.986668 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:46.986315 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2"] Oct 30 00:56:47.039781 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.039745 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" event={"ID":"4d4b60c4-d140-43ee-9c61-802cba188d6b","Type":"ContainerStarted","Data":"45a3e79d4a6cf0cb9420cdda1ae452e7894d087f35c7203f18c216f044161f6e"} Oct 30 00:56:47.040175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.039813 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" event={"ID":"4d4b60c4-d140-43ee-9c61-802cba188d6b","Type":"ContainerStarted","Data":"ab5bfa7f23231330fcad2ec2726098fc3d7559d9fa9d90e068cd33392f4a3c1a"} Oct 30 00:56:47.044936 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.043062 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" event={"ID":"80abb66d-7e8c-4524-8902-e4afd8221305","Type":"ContainerStarted","Data":"3354cd9a48775cbfe4a1b36eaf5b142b515da913776bb5cbcc66049c63303ba6"} Oct 30 00:56:47.091733 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.091678 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podStartSLOduration=2.091658697 podStartE2EDuration="2.091658697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:47.085428155 +0000 UTC m=+1092.028601347" watchObservedRunningTime="2026-10-30 00:56:47.091658697 +0000 UTC m=+1092.034831890" Oct 30 00:56:47.198621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.194226 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:47.198621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.194283 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:47.202272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.201250 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:47.202272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.201333 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:47.202272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.201372 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:56:47.222866 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.222841 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc"] Oct 30 00:56:47.223103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.223089 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:56:47.332562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.328608 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:47.506260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.506222 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-xks-ca\") pod \"66f77110-832f-4cc5-bff7-ae34fbfc719a\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " Oct 30 00:56:47.506463 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.506294 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f77110-832f-4cc5-bff7-ae34fbfc719a-kube-api-access-s8mrh\" (UniqueName: \"kubernetes.io/projected/66f77110-832f-4cc5-bff7-ae34fbfc719a-kube-api-access-s8mrh\") pod \"66f77110-832f-4cc5-bff7-ae34fbfc719a\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " Oct 30 00:56:47.506636 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.506611 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-openshift-ca\") pod \"66f77110-832f-4cc5-bff7-ae34fbfc719a\" (UID: \"66f77110-832f-4cc5-bff7-ae34fbfc719a\") " Oct 30 00:56:47.507907 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.507722 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-openshift-ca" pod "66f77110-832f-4cc5-bff7-ae34fbfc719a" (UID: "66f77110-832f-4cc5-bff7-ae34fbfc719a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:47.509147 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.509125 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx"] Oct 30 00:56:47.510378 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.510358 4905 cpu_manager.go:414] "containerMap: removing container" podUID="66f77110-832f-4cc5-bff7-ae34fbfc719a" containerName="revoke-keys" Oct 30 00:56:47.510510 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.510496 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f77110-832f-4cc5-bff7-ae34fbfc719a" containerName="revoke-keys" Oct 30 00:56:47.510830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.510815 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f77110-832f-4cc5-bff7-ae34fbfc719a" containerName="revoke-keys" Oct 30 00:56:47.573692 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.522346 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f77110-832f-4cc5-bff7-ae34fbfc719a-kube-api-access-s8mrh" pod "66f77110-832f-4cc5-bff7-ae34fbfc719a" (UID: "66f77110-832f-4cc5-bff7-ae34fbfc719a"). InnerVolumeSpecName "kube-api-access-s8mrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:47.573692 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.522450 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-xks-ca" pod "66f77110-832f-4cc5-bff7-ae34fbfc719a" (UID: "66f77110-832f-4cc5-bff7-ae34fbfc719a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:47.573692 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.573131 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.597682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.597390 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx"] Oct 30 00:56:47.610244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.609903 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:47.610244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.609936 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f77110-832f-4cc5-bff7-ae34fbfc719a-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:47.610244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.609951 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8mrh\" (UniqueName: \"kubernetes.io/projected/66f77110-832f-4cc5-bff7-ae34fbfc719a-kube-api-access-s8mrh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:47.711122 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.711088 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.711305 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.711136 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.711385 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.711370 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7v7\" (UniqueName: \"kubernetes.io/projected/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-kube-api-access-sg7v7\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.812734 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.812696 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7v7\" (UniqueName: \"kubernetes.io/projected/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-kube-api-access-sg7v7\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.813126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.813104 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.813288 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.813273 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.814107 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.814083 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.815779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.815760 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.850585 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.850481 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7v7\" (UniqueName: \"kubernetes.io/projected/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-kube-api-access-sg7v7\") pod \"maas-api-revoke-sub-95e1b12e-fx9cx\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:47.915810 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:47.915778 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:48.113136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.111836 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" event={"ID":"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6","Type":"ContainerStarted","Data":"68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6"} Oct 30 00:56:48.113136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.111912 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" event={"ID":"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6","Type":"ContainerStarted","Data":"8e822b5e1401c1f20fe4659d1c245a7ddfe3fe978e4ea33decaea8632d8ac236"} Oct 30 00:56:48.115498 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.115475 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" event={"ID":"80abb66d-7e8c-4524-8902-e4afd8221305","Type":"ContainerStarted","Data":"cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404"} Oct 30 00:56:48.119252 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.119213 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" Oct 30 00:56:48.122627 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.122599 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk" event={"ID":"66f77110-832f-4cc5-bff7-ae34fbfc719a","Type":"ContainerDied","Data":"844a44d70783182b4f36a6d926c2e7e7bd9d6d6e1a66262774021c20e43398d8"} Oct 30 00:56:48.122744 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.122652 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="844a44d70783182b4f36a6d926c2e7e7bd9d6d6e1a66262774021c20e43398d8" Oct 30 00:56:48.208732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.200667 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:48.208732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.200717 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:48.283782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.221629 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podStartSLOduration=2.221609265 podStartE2EDuration="2.221609265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:48.177859051 +0000 UTC m=+1093.121032246" watchObservedRunningTime="2026-10-30 00:56:48.221609265 +0000 UTC m=+1093.164782457" Oct 30 00:56:48.497649 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.492652 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podStartSLOduration=2.492630984 podStartE2EDuration="2.492630984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:48.248312899 +0000 UTC m=+1093.191486092" watchObservedRunningTime="2026-10-30 00:56:48.492630984 +0000 UTC m=+1093.435804178" Oct 30 00:56:48.885593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.816493 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:56:48.894572 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.887187 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:56:48.959561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.920733 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:48.959561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.920792 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:48.959561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.929155 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:48.959561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:48.929209 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:49.222358 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.222315 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:49.239385 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.239357 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:49.239596 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.239569 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:49.301723 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.291854 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:49.396629 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.386949 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:49.442716 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.437629 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:49.510551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.478344 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:49.630599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.626387 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" event={"ID":"58aa12f4-a34b-4d0a-b34b-f5bea87b554e","Type":"ContainerStarted","Data":"917aa4c8ee816057b7cc2d9406046949b29a8b5e4889cf93ce3285b0a9f2602e"} Oct 30 00:56:49.630599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.626466 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx"] Oct 30 00:56:49.716544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.696622 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:49.716823 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.716790 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:49.738548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.718258 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:49.738869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.738831 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:49.815117 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.814707 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:56:49.858698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.858655 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:56:49.859270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.859239 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:49.859708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:49.859686 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:56:50.209305 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.204884 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:50.209305 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.204940 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:50.241556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.224028 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:56:50.241556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.224432 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:56:50.309544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.304899 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:50.309544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.305608 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:56:50.397554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.387777 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:50.397554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.388481 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:56:50.431877 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.399501 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:56:50.501604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.466623 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:56:50.501604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.467027 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:56:50.527579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.503691 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:56:50.789239 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.626293 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:50.789239 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.770877 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:50.789239 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.770973 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:56:50.834869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.815603 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:50.886555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.877985 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:56:50.943570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.933986 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:56:50.943570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.934232 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:56:50.943570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.934443 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:56:50.943570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.934731 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:50.943570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.934789 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:56:50.943570 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.934825 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:56:50.992812 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:50.992767 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.049231 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.049137 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.401154 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.320292 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.247691 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.308858 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.432295 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.445592 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.445991 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.451678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.446055 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:56:51.609663 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.471649 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.830014 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.825253 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.940317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.940259 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:56:51.957667 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.957612 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.975361 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.971582 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:51.999760 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:51.999677 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:56:52.041713 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.041677 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:52.042067 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.041950 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:52.206132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.206035 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:52.319930 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.319893 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:52.342759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.337792 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:52.342759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.337852 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:52.393473 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:56:52.393440 4905 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.146s" Oct 30 00:56:52.403256 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.403227 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" event={"ID":"58aa12f4-a34b-4d0a-b34b-f5bea87b554e","Type":"ContainerStarted","Data":"ca338d41ec01333597ea7740f0e062040df218cd478458dce45690f41ab0cdd9"} Oct 30 00:56:52.403715 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.403698 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:56:52.405041 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.401612 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:52.405195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.405176 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:52.501720 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.501670 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:52.502899 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.502867 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:56:52.522438 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.522402 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:56:52.588549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.587286 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:52.683315 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.681716 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:56:52.683315 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.682971 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:52.683315 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.683021 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:52.719892 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.709815 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:52.785597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.724691 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:52.785597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.724750 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:52.785597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:52.746686 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:53.094770 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.094677 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:53.116678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.115613 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:56:53.129143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.129111 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:56:53.197275 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.197239 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:53.526283 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.526257 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:53.541673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.524800 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:53.543052 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.526060 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:56:53.544236 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.254044 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:56:53.545349 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.536260 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:56:53.595581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.595542 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:56:53.628971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.628929 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:53.648288 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.629900 4905 generic.go:356] "Generic (PLEG): container finished" podID="58aa12f4-a34b-4d0a-b34b-f5bea87b554e" containerID="ca338d41ec01333597ea7740f0e062040df218cd478458dce45690f41ab0cdd9" exitCode=0 Oct 30 00:56:53.649368 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.629924 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" event={"ID":"58aa12f4-a34b-4d0a-b34b-f5bea87b554e","Type":"ContainerDied","Data":"ca338d41ec01333597ea7740f0e062040df218cd478458dce45690f41ab0cdd9"} Oct 30 00:56:53.705955 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.705922 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:53.706187 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.706164 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:53.706552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.706503 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:53.747655 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.747612 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:53.749467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.749439 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:53.778377 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.778284 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:53.794546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.780558 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" podStartSLOduration=6.780539567 podStartE2EDuration="6.780539567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:52.380512214 +0000 UTC m=+1097.323685417" watchObservedRunningTime="2026-10-30 00:56:53.780539567 +0000 UTC m=+1098.723712759" Oct 30 00:56:53.892622 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.871662 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:53.892622 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:53.871769 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 00:56:54.130591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.100278 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:56:54.133328 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.130834 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:54.133328 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.131100 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:54.164559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.149651 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:56:54.164559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.149709 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:54.224551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.222899 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:54.224551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.222959 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:54.246561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.239850 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:56:54.246561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.239915 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:54.548110 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.548067 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:54.548904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.548877 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:56:54.549204 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.549182 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:56:54.549503 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.549482 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:56:54.604544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.601679 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:56:54.655124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.655090 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:54.770699 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.770644 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:54.770991 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.770965 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:54.772204 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.708914 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:56:54.772412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.772399 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:54.772543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.687646 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:54.772659 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.772641 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:54.772776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.732970 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:56:54.938025 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:54.937294 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:56:55.246625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.229632 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:56:55.246625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.230278 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:55.246625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.230310 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:55.619070 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.597008 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:55.624302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.620359 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:55.624302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.620664 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:55.624302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.621896 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:56:55.638905 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.638874 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:55.639129 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.639106 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:55.824560 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.823899 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:55.867604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:55.867288 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:56.184158 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.165780 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" podUID="34ba07ff-2d49-4483-a644-04ce768e4937" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 00:56:56.320132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.167073 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 00:56:56.320366 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.197586 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:56.320778 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.239245 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:56.372501 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.372472 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:56:56.411601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.410595 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:56.413086 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.413055 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:56.622663 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.622612 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:56.665890 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.623220 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:56.666454 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.624223 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:56.666650 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.666608 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:56.666776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.632809 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:56.691540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.643664 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:56.756643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.754534 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:56.756643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.701224 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:56.776478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.775293 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:56.776478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.775351 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:56.790573 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.784724 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:56:56.806719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.806687 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:56.809080 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.809026 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:56:56.823543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.812706 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:56:56.825149 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.825104 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:56.891600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:56.891482 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:57.137545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.128056 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-kube-api-access-sg7v7\" (UniqueName: \"kubernetes.io/projected/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-kube-api-access-sg7v7\") pod \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " Oct 30 00:56:57.159442 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.159353 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-openshift-ca\") pod \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " Oct 30 00:56:57.316845 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.315756 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-xks-ca\") pod \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\" (UID: \"58aa12f4-a34b-4d0a-b34b-f5bea87b554e\") " Oct 30 00:56:57.343543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.178926 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-openshift-ca" pod "58aa12f4-a34b-4d0a-b34b-f5bea87b554e" (UID: "58aa12f4-a34b-4d0a-b34b-f5bea87b554e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:57.343543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.273449 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:57.343543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.329655 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:57.343543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.329077 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-kube-api-access-sg7v7" pod "58aa12f4-a34b-4d0a-b34b-f5bea87b554e" (UID: "58aa12f4-a34b-4d0a-b34b-f5bea87b554e"). InnerVolumeSpecName "kube-api-access-sg7v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:57.466993 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.466900 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg7v7\" (UniqueName: \"kubernetes.io/projected/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-kube-api-access-sg7v7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:57.524920 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.524876 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-xks-ca" pod "58aa12f4-a34b-4d0a-b34b-f5bea87b554e" (UID: "58aa12f4-a34b-4d0a-b34b-f5bea87b554e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:56:57.537635 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.537607 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:57.631946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.630180 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:57.657621 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.645660 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:57.657856 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.657829 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:57.657966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.646216 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:57.658059 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.658041 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:57.658335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.645933 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:57.674840 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.674800 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:57.680490 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.676634 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:56:57.757975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.757937 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa12f4-a34b-4d0a-b34b-f5bea87b554e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:56:57.851161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.851122 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:57.900553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.894307 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:57.903317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.903276 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:57.903895 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:57.903874 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:58.200282 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.200164 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:58.251538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.225460 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:58.266683 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.266638 4905 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:56:58.267978 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.267922 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" Oct 30 00:56:58.274333 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.273930 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx" event={"ID":"58aa12f4-a34b-4d0a-b34b-f5bea87b554e","Type":"ContainerDied","Data":"917aa4c8ee816057b7cc2d9406046949b29a8b5e4889cf93ce3285b0a9f2602e"} Oct 30 00:56:58.274333 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.273985 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917aa4c8ee816057b7cc2d9406046949b29a8b5e4889cf93ce3285b0a9f2602e" Oct 30 00:56:58.281476 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.281405 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:58.281476 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.281455 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:58.611544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.568700 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:58.611544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.570511 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:56:58.618842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.618601 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:58.649661 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.649541 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:58.649661 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.649595 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:58.649661 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.649608 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:58.650250 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.649667 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:58.677819 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.675267 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:58.677819 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.676372 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:56:58.677819 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.677309 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:58.712474 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.708722 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:56:58.798141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.798113 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:56:58.899406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.898767 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:56:58.915538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.915036 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:56:58.926738 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:58.925927 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:56:59.076928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.076888 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl"] Oct 30 00:56:59.080079 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.080055 4905 cpu_manager.go:414] "containerMap: removing container" podUID="58aa12f4-a34b-4d0a-b34b-f5bea87b554e" containerName="revoke-keys" Oct 30 00:56:59.080229 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.080216 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aa12f4-a34b-4d0a-b34b-f5bea87b554e" containerName="revoke-keys" Oct 30 00:56:59.129543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.088007 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aa12f4-a34b-4d0a-b34b-f5bea87b554e" containerName="revoke-keys" Oct 30 00:56:59.176836 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.169320 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.185197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.183582 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl"] Oct 30 00:56:59.195800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.190792 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:56:59.195800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.191641 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:56:59.195800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.191776 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:56:59.195800 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.191854 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:56:59.196553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.196500 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:56:59.196730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.196705 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:56:59.341605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.341573 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.341947 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.341928 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.342147 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.342132 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rx7\" (UniqueName: \"kubernetes.io/projected/61bef333-81be-4014-9e2e-165b1ce2b00c-kube-api-access-f4rx7\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.443923 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.443819 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.445161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.444967 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rx7\" (UniqueName: \"kubernetes.io/projected/61bef333-81be-4014-9e2e-165b1ce2b00c-kube-api-access-f4rx7\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.445161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.445132 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.445676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.445652 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.452071 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.450660 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.463891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.463869 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn"] Oct 30 00:56:59.467119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.467089 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rx7\" (UniqueName: \"kubernetes.io/projected/61bef333-81be-4014-9e2e-165b1ce2b00c-kube-api-access-f4rx7\") pod \"maas-api-revoke-sub-b2e06912-5bnkl\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.470300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.470268 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.483548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.483504 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn"] Oct 30 00:56:59.509833 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.509812 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:56:59.558055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.557118 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.558055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.557370 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvxj\" (UniqueName: \"kubernetes.io/projected/3ba548d1-84d4-42ea-b1ce-4def93eadc01-kube-api-access-9cvxj\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.558055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.557491 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.625594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.625094 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:56:59.625594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.625191 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:56:59.648461 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.648424 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:56:59.648633 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.648484 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:56:59.666512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.664903 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.666512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.665067 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvxj\" (UniqueName: \"kubernetes.io/projected/3ba548d1-84d4-42ea-b1ce-4def93eadc01-kube-api-access-9cvxj\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.666512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.665170 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.666512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.666097 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.683674 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.680273 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.693342 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.693278 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvxj\" (UniqueName: \"kubernetes.io/projected/3ba548d1-84d4-42ea-b1ce-4def93eadc01-kube-api-access-9cvxj\") pod \"maas-api-revoke-sub-9022e55a-xb6hn\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.817023 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.816954 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:56:59.871117 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:56:59.865806 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl"] Oct 30 00:57:00.307076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:00.292663 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn"] Oct 30 00:57:00.351784 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:00.350583 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba548d1_84d4_42ea_b1ce_4def93eadc01.slice/crio-fc28918f38ec66491a8e4f492dd31b58c52c43f15c2b685d04ebb33ebb87c1d7 WatchSource:0}: Error finding container fc28918f38ec66491a8e4f492dd31b58c52c43f15c2b685d04ebb33ebb87c1d7: Status 404 returned error can't find the container with id fc28918f38ec66491a8e4f492dd31b58c52c43f15c2b685d04ebb33ebb87c1d7 Oct 30 00:57:00.387357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:00.387076 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" event={"ID":"61bef333-81be-4014-9e2e-165b1ce2b00c","Type":"ContainerStarted","Data":"cdd1aaba5d70a7c49c5874a80cb0e9f0d770888963302e044ce5f6eacbb46166"} Oct 30 00:57:01.201921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.201875 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:57:01.315703 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.315618 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:01.315703 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.315666 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:01.415542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.412077 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" event={"ID":"3ba548d1-84d4-42ea-b1ce-4def93eadc01","Type":"ContainerStarted","Data":"545a37e73956389778854390d30df7881833ee68384ab15cb1823d8888e166e5"} Oct 30 00:57:01.415542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.412139 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" event={"ID":"3ba548d1-84d4-42ea-b1ce-4def93eadc01","Type":"ContainerStarted","Data":"fc28918f38ec66491a8e4f492dd31b58c52c43f15c2b685d04ebb33ebb87c1d7"} Oct 30 00:57:01.415542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.414781 4905 generic.go:356] "Generic (PLEG): container finished" podID="61bef333-81be-4014-9e2e-165b1ce2b00c" containerID="116327becb15668b34ec3bf8c501e3f469f460058efdb2c8d639202aec573f58" exitCode=0 Oct 30 00:57:01.415542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.414901 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" event={"ID":"61bef333-81be-4014-9e2e-165b1ce2b00c","Type":"ContainerDied","Data":"116327becb15668b34ec3bf8c501e3f469f460058efdb2c8d639202aec573f58"} Oct 30 00:57:01.481543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.481081 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" podStartSLOduration=2.481048444 podStartE2EDuration="2.481048444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:01.471966758 +0000 UTC m=+1106.415139949" watchObservedRunningTime="2026-10-30 00:57:01.481048444 +0000 UTC m=+1106.424221640" Oct 30 00:57:01.636545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.624579 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:57:01.658543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.657618 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:57:01.680558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.672605 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:01.680558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.672660 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:01.681068 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.681026 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:01.681277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.681255 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:01.764419 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.763320 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk"] Oct 30 00:57:01.818543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.796505 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk"] Oct 30 00:57:01.818543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:01.796651 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.240609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.240571 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.241295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.241268 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.241463 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.241446 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/490cea2b-7c15-4d77-b163-6f0bbe49982d-kube-api-access-89pkm\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.341543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.330763 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:02.341543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.330822 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:02.365599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.361910 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:57:02.371549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.369822 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.371549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.369879 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/490cea2b-7c15-4d77-b163-6f0bbe49982d-kube-api-access-89pkm\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.371549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.370138 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.371549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.371541 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.380009 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.379864 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 00:57:02.395685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.390970 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.395685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.394279 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/490cea2b-7c15-4d77-b163-6f0bbe49982d-kube-api-access-89pkm\") pod \"maas-api-revoke-sub-16b8365f-nkqtk\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.424303 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.424277 4905 generic.go:356] "Generic (PLEG): container finished" podID="3ba548d1-84d4-42ea-b1ce-4def93eadc01" containerID="545a37e73956389778854390d30df7881833ee68384ab15cb1823d8888e166e5" exitCode=0 Oct 30 00:57:02.434069 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.425395 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" event={"ID":"3ba548d1-84d4-42ea-b1ce-4def93eadc01","Type":"ContainerDied","Data":"545a37e73956389778854390d30df7881833ee68384ab15cb1823d8888e166e5"} Oct 30 00:57:02.547739 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.538204 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:02.684182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.683799 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:02.684182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.683863 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:02.720203 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.716736 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:02.720203 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.716799 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:02.780050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.770804 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:57:02.788575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.784836 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 00:57:02.876631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.876464 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:57:02.880742 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.880685 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 00:57:02.953110 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:02.953020 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:57:03.108623 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.108597 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-xks-ca\") pod \"61bef333-81be-4014-9e2e-165b1ce2b00c\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " Oct 30 00:57:03.108802 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.108787 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-openshift-ca\") pod \"61bef333-81be-4014-9e2e-165b1ce2b00c\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " Oct 30 00:57:03.108869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.108853 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bef333-81be-4014-9e2e-165b1ce2b00c-kube-api-access-f4rx7\" (UniqueName: \"kubernetes.io/projected/61bef333-81be-4014-9e2e-165b1ce2b00c-kube-api-access-f4rx7\") pod \"61bef333-81be-4014-9e2e-165b1ce2b00c\" (UID: \"61bef333-81be-4014-9e2e-165b1ce2b00c\") " Oct 30 00:57:03.109136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.109112 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-openshift-ca" pod "61bef333-81be-4014-9e2e-165b1ce2b00c" (UID: "61bef333-81be-4014-9e2e-165b1ce2b00c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:03.109406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.109376 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:03.110967 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.110945 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-xks-ca" pod "61bef333-81be-4014-9e2e-165b1ce2b00c" (UID: "61bef333-81be-4014-9e2e-165b1ce2b00c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:03.111065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.111002 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bef333-81be-4014-9e2e-165b1ce2b00c-kube-api-access-f4rx7" pod "61bef333-81be-4014-9e2e-165b1ce2b00c" (UID: "61bef333-81be-4014-9e2e-165b1ce2b00c"). InnerVolumeSpecName "kube-api-access-f4rx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:03.201958 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:03.201929 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490cea2b_7c15_4d77_b163_6f0bbe49982d.slice/crio-e21439f1fad93baaba77076479e235581d59aeb20a5dfca6a3001603a005d9ab WatchSource:0}: Error finding container e21439f1fad93baaba77076479e235581d59aeb20a5dfca6a3001603a005d9ab: Status 404 returned error can't find the container with id e21439f1fad93baaba77076479e235581d59aeb20a5dfca6a3001603a005d9ab Oct 30 00:57:03.205422 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.205395 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk"] Oct 30 00:57:03.211301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.211278 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bef333-81be-4014-9e2e-165b1ce2b00c-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:03.211380 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.211303 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rx7\" (UniqueName: \"kubernetes.io/projected/61bef333-81be-4014-9e2e-165b1ce2b00c-kube-api-access-f4rx7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:03.434415 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.434376 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" event={"ID":"490cea2b-7c15-4d77-b163-6f0bbe49982d","Type":"ContainerStarted","Data":"e21439f1fad93baaba77076479e235581d59aeb20a5dfca6a3001603a005d9ab"} Oct 30 00:57:03.436053 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.436026 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" Oct 30 00:57:03.436194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.436047 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl" event={"ID":"61bef333-81be-4014-9e2e-165b1ce2b00c","Type":"ContainerDied","Data":"cdd1aaba5d70a7c49c5874a80cb0e9f0d770888963302e044ce5f6eacbb46166"} Oct 30 00:57:03.436194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.436103 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd1aaba5d70a7c49c5874a80cb0e9f0d770888963302e044ce5f6eacbb46166" Oct 30 00:57:03.619776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.619741 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:57:03.721552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.721240 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-xks-ca\") pod \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " Oct 30 00:57:03.721552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.721336 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba548d1-84d4-42ea-b1ce-4def93eadc01-kube-api-access-9cvxj\" (UniqueName: \"kubernetes.io/projected/3ba548d1-84d4-42ea-b1ce-4def93eadc01-kube-api-access-9cvxj\") pod \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " Oct 30 00:57:03.721552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.721425 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-openshift-ca\") pod \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\" (UID: \"3ba548d1-84d4-42ea-b1ce-4def93eadc01\") " Oct 30 00:57:03.742597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.737720 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-openshift-ca" pod "3ba548d1-84d4-42ea-b1ce-4def93eadc01" (UID: "3ba548d1-84d4-42ea-b1ce-4def93eadc01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:03.836731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.811032 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5"] Oct 30 00:57:03.836731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.812354 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3ba548d1-84d4-42ea-b1ce-4def93eadc01" containerName="revoke-keys" Oct 30 00:57:03.836731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.812375 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba548d1-84d4-42ea-b1ce-4def93eadc01" containerName="revoke-keys" Oct 30 00:57:03.836731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.812431 4905 cpu_manager.go:414] "containerMap: removing container" podUID="61bef333-81be-4014-9e2e-165b1ce2b00c" containerName="revoke-keys" Oct 30 00:57:03.836731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.812438 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bef333-81be-4014-9e2e-165b1ce2b00c" containerName="revoke-keys" Oct 30 00:57:03.836731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.812626 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bef333-81be-4014-9e2e-165b1ce2b00c" containerName="revoke-keys" Oct 30 00:57:03.836731 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.812652 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba548d1-84d4-42ea-b1ce-4def93eadc01" containerName="revoke-keys" Oct 30 00:57:03.837509 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.836892 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:03.860766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.858747 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba548d1-84d4-42ea-b1ce-4def93eadc01-kube-api-access-9cvxj" pod "3ba548d1-84d4-42ea-b1ce-4def93eadc01" (UID: "3ba548d1-84d4-42ea-b1ce-4def93eadc01"). InnerVolumeSpecName "kube-api-access-9cvxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:03.870547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.865832 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-xks-ca" pod "3ba548d1-84d4-42ea-b1ce-4def93eadc01" (UID: "3ba548d1-84d4-42ea-b1ce-4def93eadc01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:03.972540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.944059 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba548d1-84d4-42ea-b1ce-4def93eadc01-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:03.972540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:03.944099 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvxj\" (UniqueName: \"kubernetes.io/projected/3ba548d1-84d4-42ea-b1ce-4def93eadc01-kube-api-access-9cvxj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:04.051591 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.031979 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5"] Oct 30 00:57:04.051903 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.040720 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.159982 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.159593 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.160244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.160223 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.160412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.160396 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8wvm\" (UniqueName: \"kubernetes.io/projected/bd0ef1c2-a60c-4c24-a809-f02f8130a864-kube-api-access-w8wvm\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.332619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.264617 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.332619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.264718 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.332619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.264829 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8wvm\" (UniqueName: \"kubernetes.io/projected/bd0ef1c2-a60c-4c24-a809-f02f8130a864-kube-api-access-w8wvm\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.332619 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.266343 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.512544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.442896 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.512544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.456346 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8wvm\" (UniqueName: \"kubernetes.io/projected/bd0ef1c2-a60c-4c24-a809-f02f8130a864-kube-api-access-w8wvm\") pod \"maas-api-revoke-sub-50bd9dff-7gbt5\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.552155 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.552118 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:04.830550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.822070 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" event={"ID":"3ba548d1-84d4-42ea-b1ce-4def93eadc01","Type":"ContainerDied","Data":"fc28918f38ec66491a8e4f492dd31b58c52c43f15c2b685d04ebb33ebb87c1d7"} Oct 30 00:57:04.830875 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.830851 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc28918f38ec66491a8e4f492dd31b58c52c43f15c2b685d04ebb33ebb87c1d7" Oct 30 00:57:04.831079 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.831066 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn" Oct 30 00:57:04.919761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.917897 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" event={"ID":"490cea2b-7c15-4d77-b163-6f0bbe49982d","Type":"ContainerStarted","Data":"575b734ce5a6c01c9bbdabf5165a8cc5510502d42d10d5672337c7273740b7df"} Oct 30 00:57:04.992548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:04.987040 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" podStartSLOduration=3.987017796 podStartE2EDuration="3.987017796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:04.983194336 +0000 UTC m=+1109.926367530" watchObservedRunningTime="2026-10-30 00:57:04.987017796 +0000 UTC m=+1109.930190987" Oct 30 00:57:05.151598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.147863 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:05.505685 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.505636 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:05.568478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.568435 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:57:05.719546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.718609 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:05.719546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.719032 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:57:05.723538 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.722593 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:05.847118 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.844178 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:05.975720 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:05.975678 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.319984 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.220088 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.319984 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.232646 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.353184 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.353152 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:57:06.516546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.508626 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.516546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.509262 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.568512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.509533 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:57:06.568512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.547977 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.569888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.569850 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:57:06.577816 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.577713 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:57:06.788909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.764936 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:57:06.788909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.765397 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:57:06.819777 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.789785 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.819777 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.808116 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.852743 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.852623 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:06.943004 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.942974 4905 generic.go:356] "Generic (PLEG): container finished" podID="490cea2b-7c15-4d77-b163-6f0bbe49982d" containerID="575b734ce5a6c01c9bbdabf5165a8cc5510502d42d10d5672337c7273740b7df" exitCode=0 Oct 30 00:57:07.045590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.989060 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:57:07.045590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.989114 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" event={"ID":"490cea2b-7c15-4d77-b163-6f0bbe49982d","Type":"ContainerDied","Data":"575b734ce5a6c01c9bbdabf5165a8cc5510502d42d10d5672337c7273740b7df"} Oct 30 00:57:07.045590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:06.989231 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:57:07.308018 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.307880 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:07.443498 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.333477 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:57:07.472914 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.414285 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:07.527869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.516472 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:57:07.527869 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.517815 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:07.556552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.552825 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:07.576399 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.575303 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:07.778542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.775629 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:07.969546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.969001 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:07.976581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:07.971290 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:08.176909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.141219 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:57:08.176909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.141286 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:57:08.176909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.141307 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5"] Oct 30 00:57:08.176909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.141377 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:57:08.176909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.141413 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:57:08.484818 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.482096 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:08.500512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.489636 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:08.500512 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.490414 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:08.643201 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.643149 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:08.686027 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.675742 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:08.686027 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.676216 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:57:08.847081 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.821295 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:08.847081 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:08.822665 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.123640 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.022268 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.123640 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.023442 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:57:09.276916 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.276881 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:09.277186 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.277163 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.391271 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.383256 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:09.391271 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.383318 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.522562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.512209 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:09.740600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.731425 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:57:09.756339 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.756297 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:57:09.788586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.788512 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:09.788849 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.788812 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.848551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.847135 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" podUID="199f2072-d9ae-4009-9502-c6c035977c44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.874558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.865800 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.874558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.866067 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.875873 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.875828 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:57:09.877274 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.877246 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:09.878085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.878063 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:57:09.997789 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:09.997690 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" event={"ID":"bd0ef1c2-a60c-4c24-a809-f02f8130a864","Type":"ContainerStarted","Data":"f8112a87e12b9d818eb26e1287d72d1d6f335a1d3f379e56c712d5e7bdebb2ac"} Oct 30 00:57:10.026758 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.026700 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:10.027421 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.027379 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:10.558588 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.550995 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:10.579892 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:57:10.579803 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0ef1c2_a60c_4c24_a809_f02f8130a864.slice/crio-conmon-fb10886d88e2025843d16183dfced3812bd7578d698231fe0b513ba7cf05310c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0ef1c2_a60c_4c24_a809_f02f8130a864.slice/crio-fb10886d88e2025843d16183dfced3812bd7578d698231fe0b513ba7cf05310c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 00:57:10.736597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.731544 4905 generic.go:356] "Generic (PLEG): container finished" podID="bd0ef1c2-a60c-4c24-a809-f02f8130a864" containerID="fb10886d88e2025843d16183dfced3812bd7578d698231fe0b513ba7cf05310c" exitCode=0 Oct 30 00:57:10.736597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.731739 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" event={"ID":"bd0ef1c2-a60c-4c24-a809-f02f8130a864","Type":"ContainerDied","Data":"fb10886d88e2025843d16183dfced3812bd7578d698231fe0b513ba7cf05310c"} Oct 30 00:57:10.736597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.731901 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:10.763544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.762051 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:10.867820 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.867779 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:10.881986 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.881946 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:10.882513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.882485 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:57:10.882960 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.882933 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:10.895408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.888582 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:10.983409 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:10.982629 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:11.067614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.061576 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:11.067614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.062017 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:57:11.360076 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.360048 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:57:11.360413 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.360394 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:57:11.360596 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.360582 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:57:11.360762 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.360749 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:57:11.380533 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.375003 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:57:11.380533 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.375797 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:57:11.380693 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.380580 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:57:11.383729 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.383705 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:57:11.388226 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.385961 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:57:11.391217 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.391201 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:11.402101 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.402000 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:57:11.482388 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.482348 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-xks-ca\") pod \"490cea2b-7c15-4d77-b163-6f0bbe49982d\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " Oct 30 00:57:11.482589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.482450 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-openshift-ca\") pod \"490cea2b-7c15-4d77-b163-6f0bbe49982d\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " Oct 30 00:57:11.482589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.482504 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490cea2b-7c15-4d77-b163-6f0bbe49982d-kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/490cea2b-7c15-4d77-b163-6f0bbe49982d-kube-api-access-89pkm\") pod \"490cea2b-7c15-4d77-b163-6f0bbe49982d\" (UID: \"490cea2b-7c15-4d77-b163-6f0bbe49982d\") " Oct 30 00:57:11.482865 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.482834 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-openshift-ca" pod "490cea2b-7c15-4d77-b163-6f0bbe49982d" (UID: "490cea2b-7c15-4d77-b163-6f0bbe49982d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:11.484047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.484025 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:11.484761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.484737 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-xks-ca" pod "490cea2b-7c15-4d77-b163-6f0bbe49982d" (UID: "490cea2b-7c15-4d77-b163-6f0bbe49982d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:11.485018 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.484990 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490cea2b-7c15-4d77-b163-6f0bbe49982d-kube-api-access-89pkm" pod "490cea2b-7c15-4d77-b163-6f0bbe49982d" (UID: "490cea2b-7c15-4d77-b163-6f0bbe49982d"). InnerVolumeSpecName "kube-api-access-89pkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:11.585430 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.585335 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490cea2b-7c15-4d77-b163-6f0bbe49982d-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:11.585430 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.585374 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/490cea2b-7c15-4d77-b163-6f0bbe49982d-kube-api-access-89pkm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:11.746033 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.746000 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" Oct 30 00:57:11.746285 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.746032 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk" event={"ID":"490cea2b-7c15-4d77-b163-6f0bbe49982d","Type":"ContainerDied","Data":"e21439f1fad93baaba77076479e235581d59aeb20a5dfca6a3001603a005d9ab"} Oct 30 00:57:11.746285 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:11.746089 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e21439f1fad93baaba77076479e235581d59aeb20a5dfca6a3001603a005d9ab" Oct 30 00:57:12.137151 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.137124 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:12.223035 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.223000 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-xks-ca\") pod \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " Oct 30 00:57:12.223219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.223118 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0ef1c2-a60c-4c24-a809-f02f8130a864-kube-api-access-w8wvm\" (UniqueName: \"kubernetes.io/projected/bd0ef1c2-a60c-4c24-a809-f02f8130a864-kube-api-access-w8wvm\") pod \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " Oct 30 00:57:12.223420 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.223394 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-openshift-ca\") pod \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\" (UID: \"bd0ef1c2-a60c-4c24-a809-f02f8130a864\") " Oct 30 00:57:12.223871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.223845 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-openshift-ca" pod "bd0ef1c2-a60c-4c24-a809-f02f8130a864" (UID: "bd0ef1c2-a60c-4c24-a809-f02f8130a864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:12.224155 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.224131 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:12.225334 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.225307 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-xks-ca" pod "bd0ef1c2-a60c-4c24-a809-f02f8130a864" (UID: "bd0ef1c2-a60c-4c24-a809-f02f8130a864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:12.225756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.225736 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0ef1c2-a60c-4c24-a809-f02f8130a864-kube-api-access-w8wvm" pod "bd0ef1c2-a60c-4c24-a809-f02f8130a864" (UID: "bd0ef1c2-a60c-4c24-a809-f02f8130a864"). InnerVolumeSpecName "kube-api-access-w8wvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:12.325859 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.325827 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0ef1c2-a60c-4c24-a809-f02f8130a864-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:12.325859 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.325860 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8wvm\" (UniqueName: \"kubernetes.io/projected/bd0ef1c2-a60c-4c24-a809-f02f8130a864-kube-api-access-w8wvm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:12.415078 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:57:12.415037 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:57:12.757492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.757461 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" Oct 30 00:57:12.757492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.757477 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5" event={"ID":"bd0ef1c2-a60c-4c24-a809-f02f8130a864","Type":"ContainerDied","Data":"f8112a87e12b9d818eb26e1287d72d1d6f335a1d3f379e56c712d5e7bdebb2ac"} Oct 30 00:57:12.757730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:12.757551 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8112a87e12b9d818eb26e1287d72d1d6f335a1d3f379e56c712d5e7bdebb2ac" Oct 30 00:57:13.359032 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.358998 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m"] Oct 30 00:57:13.359992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.359971 4905 cpu_manager.go:414] "containerMap: removing container" podUID="490cea2b-7c15-4d77-b163-6f0bbe49982d" containerName="revoke-keys" Oct 30 00:57:13.360056 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.359996 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490cea2b-7c15-4d77-b163-6f0bbe49982d" containerName="revoke-keys" Oct 30 00:57:13.360093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.360065 4905 cpu_manager.go:414] "containerMap: removing container" podUID="bd0ef1c2-a60c-4c24-a809-f02f8130a864" containerName="revoke-keys" Oct 30 00:57:13.360093 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.360073 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0ef1c2-a60c-4c24-a809-f02f8130a864" containerName="revoke-keys" Oct 30 00:57:13.360229 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.360218 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="490cea2b-7c15-4d77-b163-6f0bbe49982d" containerName="revoke-keys" Oct 30 00:57:13.360263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.360246 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0ef1c2-a60c-4c24-a809-f02f8130a864" containerName="revoke-keys" Oct 30 00:57:13.364833 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.364813 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.379987 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.379966 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m"] Oct 30 00:57:13.380958 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.380938 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:57:13.381103 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.381089 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:57:13.390553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.387544 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:57:13.390553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.387631 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:57:13.443265 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.443224 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.443449 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.443337 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcrg\" (UniqueName: \"kubernetes.io/projected/086bd386-9177-4c16-835b-1adbef7553b4-kube-api-access-kfcrg\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.443653 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.443633 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.545164 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.545136 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.545308 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.545209 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.545308 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.545265 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcrg\" (UniqueName: \"kubernetes.io/projected/086bd386-9177-4c16-835b-1adbef7553b4-kube-api-access-kfcrg\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.545959 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.545930 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.547747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.547718 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.566302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.566277 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcrg\" (UniqueName: \"kubernetes.io/projected/086bd386-9177-4c16-835b-1adbef7553b4-kube-api-access-kfcrg\") pod \"maas-api-revoke-sub-ab723a90-9557m\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.682094 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.682028 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:13.988649 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:13.988622 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086bd386_9177_4c16_835b_1adbef7553b4.slice/crio-1137f8b354b5be334ccb579e5d59a4bd4e3b2cacc43b04a9cecf9673e043be76 WatchSource:0}: Error finding container 1137f8b354b5be334ccb579e5d59a4bd4e3b2cacc43b04a9cecf9673e043be76: Status 404 returned error can't find the container with id 1137f8b354b5be334ccb579e5d59a4bd4e3b2cacc43b04a9cecf9673e043be76 Oct 30 00:57:13.991425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:13.991403 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m"] Oct 30 00:57:14.815314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:14.812873 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" event={"ID":"086bd386-9177-4c16-835b-1adbef7553b4","Type":"ContainerStarted","Data":"ce80954b4b49ff11a5f0b59f56a3bbc2a98692b012796dc81452a5c579cf7771"} Oct 30 00:57:14.815314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:14.812988 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" event={"ID":"086bd386-9177-4c16-835b-1adbef7553b4","Type":"ContainerStarted","Data":"1137f8b354b5be334ccb579e5d59a4bd4e3b2cacc43b04a9cecf9673e043be76"} Oct 30 00:57:14.890969 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:14.890928 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" podStartSLOduration=1.890912142 podStartE2EDuration="1.890912142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:14.859390079 +0000 UTC m=+1119.802563273" watchObservedRunningTime="2026-10-30 00:57:14.890912142 +0000 UTC m=+1119.834085333" Oct 30 00:57:15.825556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:15.825486 4905 generic.go:356] "Generic (PLEG): container finished" podID="086bd386-9177-4c16-835b-1adbef7553b4" containerID="ce80954b4b49ff11a5f0b59f56a3bbc2a98692b012796dc81452a5c579cf7771" exitCode=0 Oct 30 00:57:15.826111 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:15.825582 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" event={"ID":"086bd386-9177-4c16-835b-1adbef7553b4","Type":"ContainerDied","Data":"ce80954b4b49ff11a5f0b59f56a3bbc2a98692b012796dc81452a5c579cf7771"} Oct 30 00:57:16.152337 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.152244 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm"] Oct 30 00:57:16.156946 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.156923 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.176420 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.176390 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm"] Oct 30 00:57:16.202682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.202652 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.202858 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.202737 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrvp\" (UniqueName: \"kubernetes.io/projected/b5e96bfd-f543-4590-85bc-1c6a5597c82e-kube-api-access-hvrvp\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.202858 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.202767 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.324139 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.307136 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrvp\" (UniqueName: \"kubernetes.io/projected/b5e96bfd-f543-4590-85bc-1c6a5597c82e-kube-api-access-hvrvp\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.324139 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.307211 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.351230 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.339122 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.366544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.359578 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.368790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.368671 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrvp\" (UniqueName: \"kubernetes.io/projected/b5e96bfd-f543-4590-85bc-1c6a5597c82e-kube-api-access-hvrvp\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.379112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.374927 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f07412-djzzm\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.473643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.473562 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:16.945834 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:16.938728 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm"] Oct 30 00:57:17.722788 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:17.722399 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:17.872544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:17.870489 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-openshift-ca\") pod \"086bd386-9177-4c16-835b-1adbef7553b4\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " Oct 30 00:57:17.872544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:17.870661 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-xks-ca\") pod \"086bd386-9177-4c16-835b-1adbef7553b4\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " Oct 30 00:57:17.872544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:17.870795 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086bd386-9177-4c16-835b-1adbef7553b4-kube-api-access-kfcrg\" (UniqueName: \"kubernetes.io/projected/086bd386-9177-4c16-835b-1adbef7553b4-kube-api-access-kfcrg\") pod \"086bd386-9177-4c16-835b-1adbef7553b4\" (UID: \"086bd386-9177-4c16-835b-1adbef7553b4\") " Oct 30 00:57:17.913007 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:17.912899 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-openshift-ca" pod "086bd386-9177-4c16-835b-1adbef7553b4" (UID: "086bd386-9177-4c16-835b-1adbef7553b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:18.058544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.050551 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" event={"ID":"086bd386-9177-4c16-835b-1adbef7553b4","Type":"ContainerDied","Data":"1137f8b354b5be334ccb579e5d59a4bd4e3b2cacc43b04a9cecf9673e043be76"} Oct 30 00:57:18.058544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.050614 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1137f8b354b5be334ccb579e5d59a4bd4e3b2cacc43b04a9cecf9673e043be76" Oct 30 00:57:18.058544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.050696 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m" Oct 30 00:57:18.091545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.063143 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" event={"ID":"b5e96bfd-f543-4590-85bc-1c6a5597c82e","Type":"ContainerStarted","Data":"c3ae2efd8e7c14d50ba9431bf897216fc57a6b2c0aa6e2cbad74a2075e7374bb"} Oct 30 00:57:18.091545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.063202 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" event={"ID":"b5e96bfd-f543-4590-85bc-1c6a5597c82e","Type":"ContainerStarted","Data":"190464305a4af41224072f7f865356293364ef15c24215b90f91bf2a4a6eadff"} Oct 30 00:57:18.155822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.155787 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:18.221161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.220851 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-xks-ca" pod "086bd386-9177-4c16-835b-1adbef7553b4" (UID: "086bd386-9177-4c16-835b-1adbef7553b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:18.223163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.223009 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086bd386-9177-4c16-835b-1adbef7553b4-kube-api-access-kfcrg" pod "086bd386-9177-4c16-835b-1adbef7553b4" (UID: "086bd386-9177-4c16-835b-1adbef7553b4"). InnerVolumeSpecName "kube-api-access-kfcrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:18.285544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.282085 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086bd386-9177-4c16-835b-1adbef7553b4-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:18.285544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.282122 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfcrg\" (UniqueName: \"kubernetes.io/projected/086bd386-9177-4c16-835b-1adbef7553b4-kube-api-access-kfcrg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:18.300541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.291877 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" podStartSLOduration=2.291858533 podStartE2EDuration="2.291858533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:18.120722176 +0000 UTC m=+1123.063895368" watchObservedRunningTime="2026-10-30 00:57:18.291858533 +0000 UTC m=+1123.235031726" Oct 30 00:57:18.431195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.429370 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:18.857545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.854628 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:18.857545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.855034 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:57:18.886848 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:18.885609 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:57:19.023559 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.015618 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:19.031687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.030595 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:19.133955 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.133311 4905 generic.go:356] "Generic (PLEG): container finished" podID="b5e96bfd-f543-4590-85bc-1c6a5597c82e" containerID="c3ae2efd8e7c14d50ba9431bf897216fc57a6b2c0aa6e2cbad74a2075e7374bb" exitCode=0 Oct 30 00:57:19.133955 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.133434 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" event={"ID":"b5e96bfd-f543-4590-85bc-1c6a5597c82e","Type":"ContainerDied","Data":"c3ae2efd8e7c14d50ba9431bf897216fc57a6b2c0aa6e2cbad74a2075e7374bb"} Oct 30 00:57:19.288316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.288266 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" podUID="199f2072-d9ae-4009-9502-c6c035977c44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:19.312676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.312609 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:19.369193 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.369153 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" podUID="199f2072-d9ae-4009-9502-c6c035977c44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:19.431446 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.431352 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:57:19.494306 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.494272 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:57:19.531568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.504130 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:57:19.606543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.605848 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:57:19.704600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.703612 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:19.945545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.942265 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:57:19.971597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.946178 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:57:19.972590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.968653 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:57:20.003481 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:19.969155 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:57:20.003769 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.003753 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:57:20.003888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.003879 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:57:20.032398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.032357 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:20.032939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.032908 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:57:20.033866 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.033847 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:57:20.034040 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.034027 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:57:20.425586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.394250 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:20.425586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.398266 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:57:20.521929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.521893 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:57:20.558207 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.558041 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:20.704104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.618956 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:20.704104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.679250 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:57:20.704104 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.679333 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:57:20.733544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.705841 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:20.761343 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.759375 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:57:20.980748 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:20.980644 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:21.010593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.005615 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:21.010593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.007936 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Oct 30 00:57:21.010593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.010319 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:57:21.046705 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.042472 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:21.056665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.056617 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:21.087346 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.087312 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g" podUID="38c45736-0323-495f-8968-2a96fe77d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:21.306287 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.306161 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:57:21.316813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.316790 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:57:21.347541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.316965 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:57:21.347541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.317002 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:57:21.347541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.317032 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:57:21.347541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.317062 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:57:21.347541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.317095 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:57:21.424008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.423965 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:21.479771 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.478049 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:57:21.479771 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.478103 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:57:21.482402 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.482247 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:57:21.640616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.640582 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:21.741782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.741744 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-xks-ca\") pod \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " Oct 30 00:57:21.741918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.741854 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-openshift-ca\") pod \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " Oct 30 00:57:21.742072 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.742056 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e96bfd-f543-4590-85bc-1c6a5597c82e-kube-api-access-hvrvp\" (UniqueName: \"kubernetes.io/projected/b5e96bfd-f543-4590-85bc-1c6a5597c82e-kube-api-access-hvrvp\") pod \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\" (UID: \"b5e96bfd-f543-4590-85bc-1c6a5597c82e\") " Oct 30 00:57:21.742269 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.742241 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-openshift-ca" pod "b5e96bfd-f543-4590-85bc-1c6a5597c82e" (UID: "b5e96bfd-f543-4590-85bc-1c6a5597c82e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:21.742836 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.742812 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.744199 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.744165 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-xks-ca" pod "b5e96bfd-f543-4590-85bc-1c6a5597c82e" (UID: "b5e96bfd-f543-4590-85bc-1c6a5597c82e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:21.744645 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.744625 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e96bfd-f543-4590-85bc-1c6a5597c82e-kube-api-access-hvrvp" pod "b5e96bfd-f543-4590-85bc-1c6a5597c82e" (UID: "b5e96bfd-f543-4590-85bc-1c6a5597c82e"). InnerVolumeSpecName "kube-api-access-hvrvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:21.833005 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.832974 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" Oct 30 00:57:21.833180 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.832989 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm" event={"ID":"b5e96bfd-f543-4590-85bc-1c6a5597c82e","Type":"ContainerDied","Data":"190464305a4af41224072f7f865356293364ef15c24215b90f91bf2a4a6eadff"} Oct 30 00:57:21.833294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.833202 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190464305a4af41224072f7f865356293364ef15c24215b90f91bf2a4a6eadff" Oct 30 00:57:21.844577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.844512 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvrvp\" (UniqueName: \"kubernetes.io/projected/b5e96bfd-f543-4590-85bc-1c6a5597c82e-kube-api-access-hvrvp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.844657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:21.844581 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e96bfd-f543-4590-85bc-1c6a5597c82e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:26.464535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.464484 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj"] Oct 30 00:57:26.467398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.465352 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b5e96bfd-f543-4590-85bc-1c6a5597c82e" containerName="revoke-keys" Oct 30 00:57:26.467398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.465369 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e96bfd-f543-4590-85bc-1c6a5597c82e" containerName="revoke-keys" Oct 30 00:57:26.467398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.465388 4905 cpu_manager.go:414] "containerMap: removing container" podUID="086bd386-9177-4c16-835b-1adbef7553b4" containerName="revoke-keys" Oct 30 00:57:26.467398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.465394 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086bd386-9177-4c16-835b-1adbef7553b4" containerName="revoke-keys" Oct 30 00:57:26.467398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.465495 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e96bfd-f543-4590-85bc-1c6a5597c82e" containerName="revoke-keys" Oct 30 00:57:26.467398 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.465534 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="086bd386-9177-4c16-835b-1adbef7553b4" containerName="revoke-keys" Oct 30 00:57:26.469316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.469296 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.488412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.488380 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj"] Oct 30 00:57:26.495842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.495819 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:57:26.495962 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.495862 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:57:26.500064 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.500043 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:57:26.500391 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.500371 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:57:26.526593 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.526563 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dcwj\" (UniqueName: \"kubernetes.io/projected/9803fb11-307e-412b-b066-3cc75c834bd3-kube-api-access-5dcwj\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.526737 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.526721 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.526891 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.526870 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.628254 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.628217 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dcwj\" (UniqueName: \"kubernetes.io/projected/9803fb11-307e-412b-b066-3cc75c834bd3-kube-api-access-5dcwj\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.628439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.628310 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.628439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.628389 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.629006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.628982 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.631011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.630990 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.649384 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.649360 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dcwj\" (UniqueName: \"kubernetes.io/projected/9803fb11-307e-412b-b066-3cc75c834bd3-kube-api-access-5dcwj\") pod \"maas-api-revoke-sub-b7507093-8dhgj\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.784701 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.784670 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:26.876283 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.872557 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65"] Oct 30 00:57:26.879577 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.879101 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:26.897755 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:26.895058 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65"] Oct 30 00:57:27.037180 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.037082 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2qr\" (UniqueName: \"kubernetes.io/projected/84e8e981-01ed-4171-95f5-13be50c61f86-kube-api-access-rf2qr\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.037339 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.037252 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.037424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.037406 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.095652 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:27.095624 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9803fb11_307e_412b_b066_3cc75c834bd3.slice/crio-178cdfb12cd33cd4aab795270dc126ee7aa525c2bb1354b01cdab4d79626c628 WatchSource:0}: Error finding container 178cdfb12cd33cd4aab795270dc126ee7aa525c2bb1354b01cdab4d79626c628: Status 404 returned error can't find the container with id 178cdfb12cd33cd4aab795270dc126ee7aa525c2bb1354b01cdab4d79626c628 Oct 30 00:57:27.099281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.099256 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj"] Oct 30 00:57:27.139156 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.139126 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2qr\" (UniqueName: \"kubernetes.io/projected/84e8e981-01ed-4171-95f5-13be50c61f86-kube-api-access-rf2qr\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.139288 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.139198 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.139350 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.139283 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.139889 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.139867 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.141824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.141802 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.159673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.159647 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2qr\" (UniqueName: \"kubernetes.io/projected/84e8e981-01ed-4171-95f5-13be50c61f86-kube-api-access-rf2qr\") pod \"maas-api-revoke-sub-c22a2165-qxm65\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.202164 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.202134 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:27.508868 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:27.508806 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e8e981_01ed_4171_95f5_13be50c61f86.slice/crio-9d905a2179b498c00f46d85ff1a00612ba71f7b68d22a76b8b476f68f274d80d WatchSource:0}: Error finding container 9d905a2179b498c00f46d85ff1a00612ba71f7b68d22a76b8b476f68f274d80d: Status 404 returned error can't find the container with id 9d905a2179b498c00f46d85ff1a00612ba71f7b68d22a76b8b476f68f274d80d Oct 30 00:57:27.513311 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.513284 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65"] Oct 30 00:57:27.915852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.915808 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" event={"ID":"9803fb11-307e-412b-b066-3cc75c834bd3","Type":"ContainerStarted","Data":"a3530b4f43b671ec68880117cacc5802bbef964591ebc3620945aa5c14d9ed87"} Oct 30 00:57:27.916036 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.915872 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" event={"ID":"9803fb11-307e-412b-b066-3cc75c834bd3","Type":"ContainerStarted","Data":"178cdfb12cd33cd4aab795270dc126ee7aa525c2bb1354b01cdab4d79626c628"} Oct 30 00:57:27.917291 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.917265 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" event={"ID":"84e8e981-01ed-4171-95f5-13be50c61f86","Type":"ContainerStarted","Data":"9d905a2179b498c00f46d85ff1a00612ba71f7b68d22a76b8b476f68f274d80d"} Oct 30 00:57:27.966279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:27.966235 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" podStartSLOduration=1.966223123 podStartE2EDuration="1.966223123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:27.955885067 +0000 UTC m=+1132.899058258" watchObservedRunningTime="2026-10-30 00:57:27.966223123 +0000 UTC m=+1132.909396356" Oct 30 00:57:28.927723 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:28.927693 4905 generic.go:356] "Generic (PLEG): container finished" podID="9803fb11-307e-412b-b066-3cc75c834bd3" containerID="a3530b4f43b671ec68880117cacc5802bbef964591ebc3620945aa5c14d9ed87" exitCode=0 Oct 30 00:57:28.928175 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:28.927783 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" event={"ID":"9803fb11-307e-412b-b066-3cc75c834bd3","Type":"ContainerDied","Data":"a3530b4f43b671ec68880117cacc5802bbef964591ebc3620945aa5c14d9ed87"} Oct 30 00:57:28.929830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:28.929802 4905 generic.go:356] "Generic (PLEG): container finished" podID="84e8e981-01ed-4171-95f5-13be50c61f86" containerID="89354f0a7f28db92b341ef60b8a019e3e4ff0c002d3ffa16e55cad967b2abfc0" exitCode=0 Oct 30 00:57:28.929969 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:28.929883 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" event={"ID":"84e8e981-01ed-4171-95f5-13be50c61f86","Type":"ContainerDied","Data":"89354f0a7f28db92b341ef60b8a019e3e4ff0c002d3ffa16e55cad967b2abfc0"} Oct 30 00:57:30.174772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.174747 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:30.178917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.178896 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:30.294197 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.294164 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-xks-ca\") pod \"84e8e981-01ed-4171-95f5-13be50c61f86\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " Oct 30 00:57:30.294375 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.294217 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-openshift-ca\") pod \"9803fb11-307e-412b-b066-3cc75c834bd3\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " Oct 30 00:57:30.294375 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.294241 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-openshift-ca\") pod \"84e8e981-01ed-4171-95f5-13be50c61f86\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " Oct 30 00:57:30.294375 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.294304 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9803fb11-307e-412b-b066-3cc75c834bd3-kube-api-access-5dcwj\" (UniqueName: \"kubernetes.io/projected/9803fb11-307e-412b-b066-3cc75c834bd3-kube-api-access-5dcwj\") pod \"9803fb11-307e-412b-b066-3cc75c834bd3\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " Oct 30 00:57:30.294375 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.294350 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-xks-ca\") pod \"9803fb11-307e-412b-b066-3cc75c834bd3\" (UID: \"9803fb11-307e-412b-b066-3cc75c834bd3\") " Oct 30 00:57:30.294628 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.294466 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e8e981-01ed-4171-95f5-13be50c61f86-kube-api-access-rf2qr\" (UniqueName: \"kubernetes.io/projected/84e8e981-01ed-4171-95f5-13be50c61f86-kube-api-access-rf2qr\") pod \"84e8e981-01ed-4171-95f5-13be50c61f86\" (UID: \"84e8e981-01ed-4171-95f5-13be50c61f86\") " Oct 30 00:57:30.295209 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.295179 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-openshift-ca" pod "84e8e981-01ed-4171-95f5-13be50c61f86" (UID: "84e8e981-01ed-4171-95f5-13be50c61f86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:30.295721 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.295698 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-openshift-ca" pod "9803fb11-307e-412b-b066-3cc75c834bd3" (UID: "9803fb11-307e-412b-b066-3cc75c834bd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:30.296942 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.296920 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-xks-ca" pod "84e8e981-01ed-4171-95f5-13be50c61f86" (UID: "84e8e981-01ed-4171-95f5-13be50c61f86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:30.302011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.301994 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9803fb11-307e-412b-b066-3cc75c834bd3-kube-api-access-5dcwj" pod "9803fb11-307e-412b-b066-3cc75c834bd3" (UID: "9803fb11-307e-412b-b066-3cc75c834bd3"). InnerVolumeSpecName "kube-api-access-5dcwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:30.302290 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.302265 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-xks-ca" pod "9803fb11-307e-412b-b066-3cc75c834bd3" (UID: "9803fb11-307e-412b-b066-3cc75c834bd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:30.304653 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.304624 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e8e981-01ed-4171-95f5-13be50c61f86-kube-api-access-rf2qr" pod "84e8e981-01ed-4171-95f5-13be50c61f86" (UID: "84e8e981-01ed-4171-95f5-13be50c61f86"). InnerVolumeSpecName "kube-api-access-rf2qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:30.396097 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.396064 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:30.396097 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.396095 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:30.396307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.396110 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e8e981-01ed-4171-95f5-13be50c61f86-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:30.396307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.396122 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dcwj\" (UniqueName: \"kubernetes.io/projected/9803fb11-307e-412b-b066-3cc75c834bd3-kube-api-access-5dcwj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:30.396307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.396133 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9803fb11-307e-412b-b066-3cc75c834bd3-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:30.396307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.396144 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2qr\" (UniqueName: \"kubernetes.io/projected/84e8e981-01ed-4171-95f5-13be50c61f86-kube-api-access-rf2qr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:30.953509 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.953479 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" Oct 30 00:57:30.953509 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.953477 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj" event={"ID":"9803fb11-307e-412b-b066-3cc75c834bd3","Type":"ContainerDied","Data":"178cdfb12cd33cd4aab795270dc126ee7aa525c2bb1354b01cdab4d79626c628"} Oct 30 00:57:30.953798 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.953625 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="178cdfb12cd33cd4aab795270dc126ee7aa525c2bb1354b01cdab4d79626c628" Oct 30 00:57:30.955792 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.955765 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" event={"ID":"84e8e981-01ed-4171-95f5-13be50c61f86","Type":"ContainerDied","Data":"9d905a2179b498c00f46d85ff1a00612ba71f7b68d22a76b8b476f68f274d80d"} Oct 30 00:57:30.955920 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.955811 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d905a2179b498c00f46d85ff1a00612ba71f7b68d22a76b8b476f68f274d80d" Oct 30 00:57:30.955920 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:30.955827 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65" Oct 30 00:57:33.285545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.269958 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:33.285545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.270701 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:33.419565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.408837 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:33.454758 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.454698 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 00:57:33.483762 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.483716 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:33.508671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.508625 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:33.852544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.851638 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:33.864544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.862220 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 00:57:33.991547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:33.988623 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.043546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.023632 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.207598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.206675 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.270545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.269817 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.270545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.270329 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:34.274544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.270900 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:57:34.274544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.271913 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.274544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.272399 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:57:34.392546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.388856 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.420543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.409627 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.420543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.410210 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:57:34.452544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.451673 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" podUID="199f2072-d9ae-4009-9502-c6c035977c44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.607545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.543613 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:34.607545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.544048 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:57:34.607545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.563123 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:57:34.607545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:34.575539 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:57:35.234544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.222862 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:57:35.234544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.234037 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:57:35.244887 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.244842 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:57:35.644342 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.257757 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:57:35.772078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.275034 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:57:35.783131 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.284622 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:35.788565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.325910 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:35.805921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.410682 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:57:35.805921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.411660 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:35.805921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.608721 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:35.813289 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.608882 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:35.880482 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:35.879376 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:36.213022 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.212971 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:57:36.268283 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.268252 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:57:36.268594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.268575 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:57:36.268766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.268751 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:57:36.278543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.240155 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:57:36.304548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.240216 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:57:36.718233 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.718176 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:36.818544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.791660 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:36.818544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.792049 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:36.818544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.792330 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:36.818544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.794019 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:36.818544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.807613 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:36.818544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.808009 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:57:36.854553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.819122 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:57:36.854553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:36.828598 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:37.174675 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.174631 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:37.174950 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.174918 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:37.452601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.448831 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:37.481548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.469948 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:37.889035 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.888972 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:57:37.923724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.923244 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:37.954569 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.937237 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:37.992547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.969253 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:57:37.998576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.969697 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:57:38.031550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.969877 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:38.036547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:37.970040 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:38.176610 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:38.163927 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" podUID="34ba07ff-2d49-4483-a644-04ce768e4937" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:38.176610 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:38.165661 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:57:38.224548 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:57:38.200633 4905 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.011s" Oct 30 00:57:38.466719 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:38.466613 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:38.580126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:38.580087 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:38.917546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:38.890792 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:38.938901 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:38.936178 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:38.939181 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:38.939146 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.003668 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.003617 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.035669 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.035623 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.036249 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.036218 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.070544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.044546 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:39.095673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.047183 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm"] Oct 30 00:57:39.097299 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.097270 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9803fb11-307e-412b-b066-3cc75c834bd3" containerName="revoke-keys" Oct 30 00:57:39.125615 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.125576 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9803fb11-307e-412b-b066-3cc75c834bd3" containerName="revoke-keys" Oct 30 00:57:39.125839 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.125825 4905 cpu_manager.go:414] "containerMap: removing container" podUID="84e8e981-01ed-4171-95f5-13be50c61f86" containerName="revoke-keys" Oct 30 00:57:39.125964 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.125951 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e8e981-01ed-4171-95f5-13be50c61f86" containerName="revoke-keys" Oct 30 00:57:39.126305 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.126288 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9803fb11-307e-412b-b066-3cc75c834bd3" containerName="revoke-keys" Oct 30 00:57:39.126428 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.126417 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e8e981-01ed-4171-95f5-13be50c61f86" containerName="revoke-keys" Oct 30 00:57:39.200594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.171628 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.218554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.202276 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:39.218554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.202354 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.283676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.283627 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" podUID="199f2072-d9ae-4009-9502-c6c035977c44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.434668 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.434624 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fjtbm" podUID="199f2072-d9ae-4009-9502-c6c035977c44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.476829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.476743 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.483644 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.483601 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:39.512549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.506423 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.512549 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.506491 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.547119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.547085 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ndn\" (UniqueName: \"kubernetes.io/projected/f154eb95-5060-435d-8d11-bee2f83b2f10-kube-api-access-54ndn\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.575582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.567900 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:57:39.587020 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.585389 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:57:39.589067 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.589044 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:57:39.614581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.589264 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:57:39.614781 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.589932 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:57:39.614851 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.612413 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:57:39.653536 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.650797 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.653536 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.650956 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.653536 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.651926 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ndn\" (UniqueName: \"kubernetes.io/projected/f154eb95-5060-435d-8d11-bee2f83b2f10-kube-api-access-54ndn\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.654920 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.654805 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.655220 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.655203 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:57:39.655480 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.655466 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm"] Oct 30 00:57:39.656653 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.656638 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:57:39.656817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.656804 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:57:39.662772 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.662748 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:57:39.665971 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.665873 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.675561 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.675539 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:57:39.684551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.679337 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ndn\" (UniqueName: \"kubernetes.io/projected/f154eb95-5060-435d-8d11-bee2f83b2f10-kube-api-access-54ndn\") pod \"maas-api-revoke-sub-f9db09f1-2rhrm\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:39.684798 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.684775 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5"] Oct 30 00:57:39.691541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.690664 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:57:39.691541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.690991 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.720329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.718026 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5"] Oct 30 00:57:39.720628 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.720606 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:57:39.730200 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.728886 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:57:39.749394 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.748330 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:57:39.753444 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.753411 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.771608 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.769844 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.771608 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.769997 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2srj\" (UniqueName: \"kubernetes.io/projected/f734b749-d3b3-41ff-a277-28721bafc52a-kube-api-access-j2srj\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.871604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.871566 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.871935 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.871909 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2srj\" (UniqueName: \"kubernetes.io/projected/f734b749-d3b3-41ff-a277-28721bafc52a-kube-api-access-j2srj\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.872105 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.872088 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.873403 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.873375 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.876183 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.876157 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.895580 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.895000 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2srj\" (UniqueName: \"kubernetes.io/projected/f734b749-d3b3-41ff-a277-28721bafc52a-kube-api-access-j2srj\") pod \"maas-api-revoke-sub-f6b7a97b-nfnb5\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:39.949195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:39.949162 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:40.059178 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:40.056315 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:40.301865 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:40.301779 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm"] Oct 30 00:57:40.447048 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:40.447013 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf734b749_d3b3_41ff_a277_28721bafc52a.slice/crio-09fabdb797f5ffc89c8b48d87c1a00afecad0a395720f051347bdeaa4b5c7dd1 WatchSource:0}: Error finding container 09fabdb797f5ffc89c8b48d87c1a00afecad0a395720f051347bdeaa4b5c7dd1: Status 404 returned error can't find the container with id 09fabdb797f5ffc89c8b48d87c1a00afecad0a395720f051347bdeaa4b5c7dd1 Oct 30 00:57:40.452779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:40.452757 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5"] Oct 30 00:57:40.637776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:40.637687 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" event={"ID":"f154eb95-5060-435d-8d11-bee2f83b2f10","Type":"ContainerStarted","Data":"2862184163fe0887d21abc5fe1c68186d8089d38e50834269daeba91861d0f24"} Oct 30 00:57:40.639324 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:40.639294 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" event={"ID":"f734b749-d3b3-41ff-a277-28721bafc52a","Type":"ContainerStarted","Data":"09fabdb797f5ffc89c8b48d87c1a00afecad0a395720f051347bdeaa4b5c7dd1"} Oct 30 00:57:41.670929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:41.666351 4905 generic.go:356] "Generic (PLEG): container finished" podID="f154eb95-5060-435d-8d11-bee2f83b2f10" containerID="aa4029121dedbca3c03222c09c4e406dad67c84f6c74a01d83abab73323e67b2" exitCode=0 Oct 30 00:57:41.670929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:41.666561 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" event={"ID":"f154eb95-5060-435d-8d11-bee2f83b2f10","Type":"ContainerDied","Data":"aa4029121dedbca3c03222c09c4e406dad67c84f6c74a01d83abab73323e67b2"} Oct 30 00:57:41.670929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:41.669652 4905 generic.go:356] "Generic (PLEG): container finished" podID="f734b749-d3b3-41ff-a277-28721bafc52a" containerID="f0e71122fff2db57e8969b481f2e3548ca0f310d28f60d082e2a55d1c4db7064" exitCode=0 Oct 30 00:57:41.670929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:41.669925 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" event={"ID":"f734b749-d3b3-41ff-a277-28721bafc52a","Type":"ContainerDied","Data":"f0e71122fff2db57e8969b481f2e3548ca0f310d28f60d082e2a55d1c4db7064"} Oct 30 00:57:42.415438 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:57:42.415406 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:57:42.921436 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:42.921412 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:42.923767 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:42.923751 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:43.057571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.057507 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f154eb95-5060-435d-8d11-bee2f83b2f10-kube-api-access-54ndn\" (UniqueName: \"kubernetes.io/projected/f154eb95-5060-435d-8d11-bee2f83b2f10-kube-api-access-54ndn\") pod \"f154eb95-5060-435d-8d11-bee2f83b2f10\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " Oct 30 00:57:43.057756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.057620 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-openshift-ca\") pod \"f734b749-d3b3-41ff-a277-28721bafc52a\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " Oct 30 00:57:43.057756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.057678 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f734b749-d3b3-41ff-a277-28721bafc52a-kube-api-access-j2srj\" (UniqueName: \"kubernetes.io/projected/f734b749-d3b3-41ff-a277-28721bafc52a-kube-api-access-j2srj\") pod \"f734b749-d3b3-41ff-a277-28721bafc52a\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " Oct 30 00:57:43.057756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.057724 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-xks-ca\") pod \"f154eb95-5060-435d-8d11-bee2f83b2f10\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " Oct 30 00:57:43.057875 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.057789 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-openshift-ca\") pod \"f154eb95-5060-435d-8d11-bee2f83b2f10\" (UID: \"f154eb95-5060-435d-8d11-bee2f83b2f10\") " Oct 30 00:57:43.057948 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.057928 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-xks-ca\") pod \"f734b749-d3b3-41ff-a277-28721bafc52a\" (UID: \"f734b749-d3b3-41ff-a277-28721bafc52a\") " Oct 30 00:57:43.058043 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.058013 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-openshift-ca" pod "f734b749-d3b3-41ff-a277-28721bafc52a" (UID: "f734b749-d3b3-41ff-a277-28721bafc52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:43.058581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.058552 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-openshift-ca" pod "f154eb95-5060-435d-8d11-bee2f83b2f10" (UID: "f154eb95-5060-435d-8d11-bee2f83b2f10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:43.058738 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.058693 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:43.058738 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.058713 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:43.060684 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.060658 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f154eb95-5060-435d-8d11-bee2f83b2f10-kube-api-access-54ndn" pod "f154eb95-5060-435d-8d11-bee2f83b2f10" (UID: "f154eb95-5060-435d-8d11-bee2f83b2f10"). InnerVolumeSpecName "kube-api-access-54ndn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:43.060783 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.060761 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-xks-ca" pod "f734b749-d3b3-41ff-a277-28721bafc52a" (UID: "f734b749-d3b3-41ff-a277-28721bafc52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:43.060851 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.060790 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f734b749-d3b3-41ff-a277-28721bafc52a-kube-api-access-j2srj" pod "f734b749-d3b3-41ff-a277-28721bafc52a" (UID: "f734b749-d3b3-41ff-a277-28721bafc52a"). InnerVolumeSpecName "kube-api-access-j2srj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:43.061191 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.061163 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-xks-ca" pod "f154eb95-5060-435d-8d11-bee2f83b2f10" (UID: "f154eb95-5060-435d-8d11-bee2f83b2f10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:43.160258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.160226 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54ndn\" (UniqueName: \"kubernetes.io/projected/f154eb95-5060-435d-8d11-bee2f83b2f10-kube-api-access-54ndn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:43.160258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.160260 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2srj\" (UniqueName: \"kubernetes.io/projected/f734b749-d3b3-41ff-a277-28721bafc52a-kube-api-access-j2srj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:43.160456 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.160276 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f154eb95-5060-435d-8d11-bee2f83b2f10-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:43.160456 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.160288 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f734b749-d3b3-41ff-a277-28721bafc52a-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:43.358884 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.358807 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4"] Oct 30 00:57:43.359483 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.359454 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f734b749-d3b3-41ff-a277-28721bafc52a" containerName="revoke-keys" Oct 30 00:57:43.359483 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.359474 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f734b749-d3b3-41ff-a277-28721bafc52a" containerName="revoke-keys" Oct 30 00:57:43.359708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.359565 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f154eb95-5060-435d-8d11-bee2f83b2f10" containerName="revoke-keys" Oct 30 00:57:43.359708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.359574 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f154eb95-5060-435d-8d11-bee2f83b2f10" containerName="revoke-keys" Oct 30 00:57:43.359818 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.359768 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f154eb95-5060-435d-8d11-bee2f83b2f10" containerName="revoke-keys" Oct 30 00:57:43.359818 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.359791 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f734b749-d3b3-41ff-a277-28721bafc52a" containerName="revoke-keys" Oct 30 00:57:43.363909 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.363890 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.383096 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.383066 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4"] Oct 30 00:57:43.464818 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.464784 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.464994 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.464825 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjqr\" (UniqueName: \"kubernetes.io/projected/f582bd9e-7b3b-4d70-b614-03671b5eb034-kube-api-access-6xjqr\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.465090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.465071 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.566255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.566221 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.566446 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.566334 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.566446 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.566357 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjqr\" (UniqueName: \"kubernetes.io/projected/f582bd9e-7b3b-4d70-b614-03671b5eb034-kube-api-access-6xjqr\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.567015 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.566991 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.568935 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.568909 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.591127 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.591097 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjqr\" (UniqueName: \"kubernetes.io/projected/f582bd9e-7b3b-4d70-b614-03671b5eb034-kube-api-access-6xjqr\") pod \"maas-api-revoke-sub-0a94c1de-v52x4\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.681950 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.681870 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:43.690439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.690414 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" Oct 30 00:57:43.690439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.690429 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm" event={"ID":"f154eb95-5060-435d-8d11-bee2f83b2f10","Type":"ContainerDied","Data":"2862184163fe0887d21abc5fe1c68186d8089d38e50834269daeba91861d0f24"} Oct 30 00:57:43.690616 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.690486 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2862184163fe0887d21abc5fe1c68186d8089d38e50834269daeba91861d0f24" Oct 30 00:57:43.692171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.692153 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" Oct 30 00:57:43.692267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.692153 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5" event={"ID":"f734b749-d3b3-41ff-a277-28721bafc52a","Type":"ContainerDied","Data":"09fabdb797f5ffc89c8b48d87c1a00afecad0a395720f051347bdeaa4b5c7dd1"} Oct 30 00:57:43.692322 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.692279 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09fabdb797f5ffc89c8b48d87c1a00afecad0a395720f051347bdeaa4b5c7dd1" Oct 30 00:57:43.991422 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:43.991393 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf582bd9e_7b3b_4d70_b614_03671b5eb034.slice/crio-b2f8165a74d3b192a3bafc8eeefb00b05074de79e7f746a60a721643e442095b WatchSource:0}: Error finding container b2f8165a74d3b192a3bafc8eeefb00b05074de79e7f746a60a721643e442095b: Status 404 returned error can't find the container with id b2f8165a74d3b192a3bafc8eeefb00b05074de79e7f746a60a721643e442095b Oct 30 00:57:43.997024 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:43.996989 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4"] Oct 30 00:57:44.703667 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:44.703580 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" event={"ID":"f582bd9e-7b3b-4d70-b614-03671b5eb034","Type":"ContainerStarted","Data":"761f06935d00ce5a0d8b37fa062384e3a7bcea91f69b0790bffa4a0a498108d3"} Oct 30 00:57:44.703667 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:44.703644 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" event={"ID":"f582bd9e-7b3b-4d70-b614-03671b5eb034","Type":"ContainerStarted","Data":"b2f8165a74d3b192a3bafc8eeefb00b05074de79e7f746a60a721643e442095b"} Oct 30 00:57:44.752040 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:44.751969 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" podStartSLOduration=1.751954253 podStartE2EDuration="1.751954253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:44.746567474 +0000 UTC m=+1149.689740665" watchObservedRunningTime="2026-10-30 00:57:44.751954253 +0000 UTC m=+1149.695127477" Oct 30 00:57:45.713266 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.713231 4905 generic.go:356] "Generic (PLEG): container finished" podID="f582bd9e-7b3b-4d70-b614-03671b5eb034" containerID="761f06935d00ce5a0d8b37fa062384e3a7bcea91f69b0790bffa4a0a498108d3" exitCode=0 Oct 30 00:57:45.713682 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.713310 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" event={"ID":"f582bd9e-7b3b-4d70-b614-03671b5eb034","Type":"ContainerDied","Data":"761f06935d00ce5a0d8b37fa062384e3a7bcea91f69b0790bffa4a0a498108d3"} Oct 30 00:57:45.816989 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.816946 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n"] Oct 30 00:57:45.822592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.822562 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.851375 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.851342 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n"] Oct 30 00:57:45.906910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.906878 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907082 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.906953 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907135 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.907092 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907135 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.907122 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907294 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.907274 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907359 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.907322 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907405 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.907365 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907405 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.907386 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:45.907505 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:45.907445 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hqg\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-kube-api-access-h7hqg\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.009451 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009411 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.009451 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009450 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.009734 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009579 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.009734 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009611 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.009734 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009629 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hqg\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-kube-api-access-h7hqg\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.009734 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009704 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009773 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009857 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009884 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009938 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009954 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.009965 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.010103 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.010732 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.010452 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.012268 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.012248 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.012491 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.012474 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.030830 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.030807 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.032313 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.032291 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hqg\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-kube-api-access-h7hqg\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.145084 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.145046 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:46.466324 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:46.466299 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993d149f_94af_4e5c_8c20_cf4f2f1955e6.slice/crio-e0f8bdbfd59b8003973e88a00a46a149fc6930b0fdf8019734778524c8259351 WatchSource:0}: Error finding container e0f8bdbfd59b8003973e88a00a46a149fc6930b0fdf8019734778524c8259351: Status 404 returned error can't find the container with id e0f8bdbfd59b8003973e88a00a46a149fc6930b0fdf8019734778524c8259351 Oct 30 00:57:46.469033 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.468997 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:57:46.469136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.469068 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:57:46.469136 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.469121 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:57:46.470251 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.470229 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n"] Oct 30 00:57:46.725116 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.725021 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" event={"ID":"993d149f-94af-4e5c-8c20-cf4f2f1955e6","Type":"ContainerStarted","Data":"fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72"} Oct 30 00:57:46.725116 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.725082 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" event={"ID":"993d149f-94af-4e5c-8c20-cf4f2f1955e6","Type":"ContainerStarted","Data":"e0f8bdbfd59b8003973e88a00a46a149fc6930b0fdf8019734778524c8259351"} Oct 30 00:57:46.773614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.773566 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podStartSLOduration=1.773548949 podStartE2EDuration="1.773548949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:46.768941019 +0000 UTC m=+1151.712114213" watchObservedRunningTime="2026-10-30 00:57:46.773548949 +0000 UTC m=+1151.716722138" Oct 30 00:57:46.902871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:46.902845 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:47.037631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.037600 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f582bd9e-7b3b-4d70-b614-03671b5eb034-kube-api-access-6xjqr\" (UniqueName: \"kubernetes.io/projected/f582bd9e-7b3b-4d70-b614-03671b5eb034-kube-api-access-6xjqr\") pod \"f582bd9e-7b3b-4d70-b614-03671b5eb034\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " Oct 30 00:57:47.037832 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.037689 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-xks-ca\") pod \"f582bd9e-7b3b-4d70-b614-03671b5eb034\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " Oct 30 00:57:47.037910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.037855 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-openshift-ca\") pod \"f582bd9e-7b3b-4d70-b614-03671b5eb034\" (UID: \"f582bd9e-7b3b-4d70-b614-03671b5eb034\") " Oct 30 00:57:47.038242 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.038213 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-openshift-ca" pod "f582bd9e-7b3b-4d70-b614-03671b5eb034" (UID: "f582bd9e-7b3b-4d70-b614-03671b5eb034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:47.038635 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.038613 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:47.039945 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.039911 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-xks-ca" pod "f582bd9e-7b3b-4d70-b614-03671b5eb034" (UID: "f582bd9e-7b3b-4d70-b614-03671b5eb034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:57:47.040272 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.040253 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f582bd9e-7b3b-4d70-b614-03671b5eb034-kube-api-access-6xjqr" pod "f582bd9e-7b3b-4d70-b614-03671b5eb034" (UID: "f582bd9e-7b3b-4d70-b614-03671b5eb034"). InnerVolumeSpecName "kube-api-access-6xjqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:47.145907 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.145868 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:47.146116 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.145870 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjqr\" (UniqueName: \"kubernetes.io/projected/f582bd9e-7b3b-4d70-b614-03671b5eb034-kube-api-access-6xjqr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:47.146116 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.145960 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f582bd9e-7b3b-4d70-b614-03671b5eb034-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:57:47.155701 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.155655 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:57:47.155911 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.155739 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:57:47.756766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.756728 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" Oct 30 00:57:47.760266 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.757625 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4" event={"ID":"f582bd9e-7b3b-4d70-b614-03671b5eb034","Type":"ContainerDied","Data":"b2f8165a74d3b192a3bafc8eeefb00b05074de79e7f746a60a721643e442095b"} Oct 30 00:57:47.760266 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:47.757680 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f8165a74d3b192a3bafc8eeefb00b05074de79e7f746a60a721643e442095b" Oct 30 00:57:48.149037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:48.148933 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:57:48.149037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:48.148999 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:57:49.146885 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:49.146836 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:57:49.147408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:49.146936 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:57:50.153678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:50.149671 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:57:50.153678 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:50.149893 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:57:51.154543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.146932 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:57:51.154543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.147006 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:57:51.265138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.265085 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62"] Oct 30 00:57:51.271243 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.271208 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f582bd9e-7b3b-4d70-b614-03671b5eb034" containerName="revoke-keys" Oct 30 00:57:51.271457 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.271441 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f582bd9e-7b3b-4d70-b614-03671b5eb034" containerName="revoke-keys" Oct 30 00:57:51.271825 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.271807 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f582bd9e-7b3b-4d70-b614-03671b5eb034" containerName="revoke-keys" Oct 30 00:57:51.290693 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.290558 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62"] Oct 30 00:57:51.290997 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.290978 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.348038 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.348001 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-podinfo\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.348296 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-socket\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363182 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-baf5781e-openshift-default-dockercfg-zcthh\"" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363248 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-envoy\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363389 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27ph\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-kube-api-access-s27ph\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363468 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-data\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363502 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-token\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363616 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-credential-socket\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363661 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7f16c3-7f67-4664-a08d-2a088130b04b-istiod-ca-cert\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.364222 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.363833 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-certs\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.465990 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-credential-socket\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466053 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7f16c3-7f67-4664-a08d-2a088130b04b-istiod-ca-cert\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466186 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-certs\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466266 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-podinfo\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466324 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-socket\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466380 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-envoy\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466477 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27ph\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-kube-api-access-s27ph\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466560 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-data\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.466592 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-token\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.467324 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-credential-socket\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.468001 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7f16c3-7f67-4664-a08d-2a088130b04b-istiod-ca-cert\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.468252 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-certs\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.470181 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-data\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.472604 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.470198 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-socket\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.502141 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.502107 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-podinfo\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.521341 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.521267 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27ph\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-kube-api-access-s27ph\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.575547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.574357 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-envoy\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.585039 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.585004 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-token\") pod \"e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:51.623073 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:51.622801 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:52.160542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:52.160370 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:57:52.160542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:52.160433 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:57:52.380596 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:52.378391 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62"] Oct 30 00:57:52.473548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:52.442986 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:57:52.473548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:52.443087 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:57:52.473548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:52.443137 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:57:53.003551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.000935 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" event={"ID":"8e7f16c3-7f67-4664-a08d-2a088130b04b","Type":"ContainerStarted","Data":"b913b8a0e2a70f9734c304c70ef0e591df13f7d4147b76f4a7cc63844b3d74b2"} Oct 30 00:57:53.146432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.146389 4905 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:57:53.146667 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.146451 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:57:53.965575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.965205 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85"] Oct 30 00:57:53.970454 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.970086 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:53.987279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.987252 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85"] Oct 30 00:57:53.993903 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.993496 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:57:53.993903 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.993614 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:57:53.993903 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:53.993686 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:57:54.001994 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.001960 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:57:54.023122 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.023084 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" event={"ID":"8e7f16c3-7f67-4664-a08d-2a088130b04b","Type":"ContainerStarted","Data":"a6ea5984b8ffd61905ac0f0c27ce56f936fa530b26bcc2f81eb91be38e86679e"} Oct 30 00:57:54.024543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.024485 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzw6\" (UniqueName: \"kubernetes.io/projected/d761e306-1293-4a66-ab30-82658e2ab0fb-kube-api-access-ctzw6\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.024666 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.024579 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.038171 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.024773 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.076505 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.075359 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" podStartSLOduration=3.075341412 podStartE2EDuration="3.075341412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:54.068140519 +0000 UTC m=+1159.011313712" watchObservedRunningTime="2026-10-30 00:57:54.075341412 +0000 UTC m=+1159.018514606" Oct 30 00:57:54.126365 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.126331 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzw6\" (UniqueName: \"kubernetes.io/projected/d761e306-1293-4a66-ab30-82658e2ab0fb-kube-api-access-ctzw6\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.126511 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.126387 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.126630 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.126610 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.127507 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.127479 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.129100 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.129074 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.147993 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.147965 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzw6\" (UniqueName: \"kubernetes.io/projected/d761e306-1293-4a66-ab30-82658e2ab0fb-kube-api-access-ctzw6\") pod \"maas-api-revoke-sub-9116d3ed-zth85\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.304707 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.304674 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:57:54.551295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.551242 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:54.552030 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.551975 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:54.554912 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.554397 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 00:57:54.625015 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.624985 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:54.630790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.630763 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:54.695148 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:54.695104 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85"] Oct 30 00:57:55.037767 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.037727 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" event={"ID":"d761e306-1293-4a66-ab30-82658e2ab0fb","Type":"ContainerStarted","Data":"32ae88c94956cfe7e2c9284829826c53a14317c36b9199464dde8f5911d707bd"} Oct 30 00:57:55.038437 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.038410 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:55.039462 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.039440 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:57:55.543539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.543348 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz"] Oct 30 00:57:55.555543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.549508 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.611826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.607961 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4fp\" (UniqueName: \"kubernetes.io/projected/c64ea606-fc60-41d2-aa91-c7d8352e9988-kube-api-access-hl4fp\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.616545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.612433 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.640544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.636949 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.694644 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.694606 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz"] Oct 30 00:57:55.744165 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.744130 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.744406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.744289 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4fp\" (UniqueName: \"kubernetes.io/projected/c64ea606-fc60-41d2-aa91-c7d8352e9988-kube-api-access-hl4fp\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.744406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.744336 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.745427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.745397 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.747723 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.747694 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.767108 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.766594 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4fp\" (UniqueName: \"kubernetes.io/projected/c64ea606-fc60-41d2-aa91-c7d8352e9988-kube-api-access-hl4fp\") pod \"maas-api-revoke-sub-c43d0110-xlcqz\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:55.986810 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:55.986778 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:57:56.060000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:56.059967 4905 generic.go:356] "Generic (PLEG): container finished" podID="d761e306-1293-4a66-ab30-82658e2ab0fb" containerID="186a8eb8fb7aa959ef10ad71a638585b7a5d102aa104aba89395e043cc864272" exitCode=0 Oct 30 00:57:56.061724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:56.061695 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" event={"ID":"d761e306-1293-4a66-ab30-82658e2ab0fb","Type":"ContainerDied","Data":"186a8eb8fb7aa959ef10ad71a638585b7a5d102aa104aba89395e043cc864272"} Oct 30 00:57:56.295851 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:57:56.295816 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64ea606_fc60_41d2_aa91_c7d8352e9988.slice/crio-efdea88039d426ebfc8e7014638967e2a4b7488978d62145e4dcb45de14b3471 WatchSource:0}: Error finding container efdea88039d426ebfc8e7014638967e2a4b7488978d62145e4dcb45de14b3471: Status 404 returned error can't find the container with id efdea88039d426ebfc8e7014638967e2a4b7488978d62145e4dcb45de14b3471 Oct 30 00:57:56.299926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:56.299903 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz"] Oct 30 00:57:57.811671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.796575 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" event={"ID":"c64ea606-fc60-41d2-aa91-c7d8352e9988","Type":"ContainerStarted","Data":"14daf8e9b770e2a85a8c2f26540e72dda000db9f56bc95beb435ffd9519a90f3"} Oct 30 00:57:57.855319 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.855285 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" event={"ID":"c64ea606-fc60-41d2-aa91-c7d8352e9988","Type":"ContainerStarted","Data":"efdea88039d426ebfc8e7014638967e2a4b7488978d62145e4dcb45de14b3471"} Oct 30 00:57:57.855442 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.802746 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:57:57.914589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.868706 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:57.914589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.848616 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:57.914589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.802708 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:57.914589 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.886443 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq"] Oct 30 00:57:58.015652 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:57.999747 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.043715 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.041877 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a03b2130\"" Oct 30 00:57:58.043715 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.042021 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a03b2130-dockercfg-klbw8\"" Oct 30 00:57:58.186624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.157636 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e453d1-4de1-4c9a-9e56-512e305ab3b4-plugins-config-volume\") pod \"payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.198809 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.198771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq8pn\" (UniqueName: \"kubernetes.io/projected/47e453d1-4de1-4c9a-9e56-512e305ab3b4-kube-api-access-xq8pn\") pod \"payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.296051 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.295994 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:58.342552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.330478 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e453d1-4de1-4c9a-9e56-512e305ab3b4-plugins-config-volume\") pod \"payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.342552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.311376 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e453d1-4de1-4c9a-9e56-512e305ab3b4-plugins-config-volume\") pod \"payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.342552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.330865 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8pn\" (UniqueName: \"kubernetes.io/projected/47e453d1-4de1-4c9a-9e56-512e305ab3b4-kube-api-access-xq8pn\") pod \"payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.474546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.471123 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq"] Oct 30 00:57:58.571915 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.571780 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:58.813952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.801287 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq8pn\" (UniqueName: \"kubernetes.io/projected/47e453d1-4de1-4c9a-9e56-512e305ab3b4-kube-api-access-xq8pn\") pod \"payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.865743 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.832027 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:58.866639 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.866609 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5"] Oct 30 00:57:58.893694 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.893651 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:58.899544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.899477 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:58.944091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.944047 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:58.946598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.946563 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:57:58.977472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.977428 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:58.978006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.977975 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:57:58.997968 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.997931 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:57:58.998493 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:58.998458 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:57:59.168032 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.165030 4905 generic.go:356] "Generic (PLEG): container finished" podID="c64ea606-fc60-41d2-aa91-c7d8352e9988" containerID="14daf8e9b770e2a85a8c2f26540e72dda000db9f56bc95beb435ffd9519a90f3" exitCode=0 Oct 30 00:57:59.256332 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.256306 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:57:59.256677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.256604 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:57:59.303867 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.303842 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:57:59.304190 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.304171 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:57:59.322432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.322394 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" event={"ID":"c64ea606-fc60-41d2-aa91-c7d8352e9988","Type":"ContainerDied","Data":"14daf8e9b770e2a85a8c2f26540e72dda000db9f56bc95beb435ffd9519a90f3"} Oct 30 00:57:59.335546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.324334 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:59.381087 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.381047 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxpsb\" (UniqueName: \"kubernetes.io/projected/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-kube-api-access-pxpsb\") pod \"payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:57:59.436596 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.435781 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:57:59.444552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.443577 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:57:59.444552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.443643 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:57:59.483143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.483115 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5"] Oct 30 00:57:59.555555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.547192 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:57:59.555555 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.547430 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxpsb\" (UniqueName: \"kubernetes.io/projected/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-kube-api-access-pxpsb\") pod \"payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:57:59.578439 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.578397 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:59.746411 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.583477 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:57:59.746411 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.732308 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:57:59.959575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.931697 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:57:59.959575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.932246 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:57:59.959575 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.933675 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:57:59.967730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.960283 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:57:59.967730 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:57:59.963013 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:58:00.001560 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.001428 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:58:00.002374 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.002337 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:58:00.045633 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.045592 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:00.046196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.046166 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:58:00.046654 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.046626 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:00.072429 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.072396 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:58:00.072759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.072743 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:58:00.396771 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.396680 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:00.446536 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.438105 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" podStartSLOduration=5.438085602 podStartE2EDuration="5.438085602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:58.228987715 +0000 UTC m=+1163.172160908" watchObservedRunningTime="2026-10-30 00:58:00.438085602 +0000 UTC m=+1165.381258796" Oct 30 00:58:00.679693 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.640585 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxpsb\" (UniqueName: \"kubernetes.io/projected/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-kube-api-access-pxpsb\") pod \"payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:58:00.760863 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.760817 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.005601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.975571 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:58:01.005601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:00.979217 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.121028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.053385 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:58:01.121028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.053760 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:58:01.121028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.054000 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:58:01.121028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.054204 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Oct 30 00:58:01.121028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.064294 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.121028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.078628 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:58:01.121028 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.079558 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.170687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.170645 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.449601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.449554 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.450079 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.450046 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:58:01.765954 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.765534 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.939049 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.938957 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:01.993544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:01.981988 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:02.080599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.059823 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:02.080599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.061018 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:58:02.080599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.062076 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:58:02.091706 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.090596 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:58:02.092936 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.091928 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:58:02.094096 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.093106 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:02.240579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.219868 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:02.240579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.219932 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:02.240579 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.229603 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:02.365565 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.358617 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:02.675547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.669579 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:58:02.846573 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.833000 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:02.926639 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.922129 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:02.992273 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:02.992218 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:03.115737 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:03.084888 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:03.125143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:03.084977 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:58:03.125728 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:03.085020 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:58:03.127203 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:03.094666 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:03.372697 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:03.094725 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:03.372697 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:03.094748 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:03.830185 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:03.825827 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:04.273545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.269669 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm" podUID="34ba07ff-2d49-4483-a644-04ce768e4937" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:04.302588 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.298864 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.326266 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.326339 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.326370 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.326403 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.326433 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.901308 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.434265 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.346218 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.433159 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.902216 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:04.928605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.830717 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:58:04.939585 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.468670 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:04.992910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.588653 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 00:58:04.992910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.843744 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 00:58:04.992910 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:58:04.665601 4905 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.4s" Oct 30 00:58:04.992910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.843473 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:58:04.992910 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:04.843613 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:05.735212 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:05.735163 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 00:58:05.980797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:05.976409 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:58:05.984274 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:05.984233 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:05.984793 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:05.984766 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:06.013704 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.013453 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:58:06.053545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.028753 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:58:06.053545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.029084 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:58:06.053545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.029330 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:58:06.053545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.050760 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:58:06.053545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.051221 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:58:06.053545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.051558 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 00:58:06.054452 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.054402 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:58:06.055411 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.055357 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:58:06.083417 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.083376 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d761e306-1293-4a66-ab30-82658e2ab0fb-kube-api-access-ctzw6\" (UniqueName: \"kubernetes.io/projected/d761e306-1293-4a66-ab30-82658e2ab0fb-kube-api-access-ctzw6\") pod \"d761e306-1293-4a66-ab30-82658e2ab0fb\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " Oct 30 00:58:06.170546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.136605 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-xks-ca\") pod \"d761e306-1293-4a66-ab30-82658e2ab0fb\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " Oct 30 00:58:06.212841 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.212806 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-openshift-ca\") pod \"d761e306-1293-4a66-ab30-82658e2ab0fb\" (UID: \"d761e306-1293-4a66-ab30-82658e2ab0fb\") " Oct 30 00:58:06.235574 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.118001 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:06.268595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.264723 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:06.279594 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.279557 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" event={"ID":"d761e306-1293-4a66-ab30-82658e2ab0fb","Type":"ContainerDied","Data":"32ae88c94956cfe7e2c9284829826c53a14317c36b9199464dde8f5911d707bd"} Oct 30 00:58:06.283646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.283613 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32ae88c94956cfe7e2c9284829826c53a14317c36b9199464dde8f5911d707bd" Oct 30 00:58:06.283932 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.283916 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85" Oct 30 00:58:06.298761 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.297849 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-openshift-ca" pod "d761e306-1293-4a66-ab30-82658e2ab0fb" (UID: "d761e306-1293-4a66-ab30-82658e2ab0fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:06.300827 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.300799 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:58:06.329344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.327160 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:06.428788 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.333633 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-xks-ca" pod "d761e306-1293-4a66-ab30-82658e2ab0fb" (UID: "d761e306-1293-4a66-ab30-82658e2ab0fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:06.429009 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.383504 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" event={"ID":"c64ea606-fc60-41d2-aa91-c7d8352e9988","Type":"ContainerDied","Data":"efdea88039d426ebfc8e7014638967e2a4b7488978d62145e4dcb45de14b3471"} Oct 30 00:58:06.449540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.429147 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efdea88039d426ebfc8e7014638967e2a4b7488978d62145e4dcb45de14b3471" Oct 30 00:58:06.482797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.482765 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" event={"ID":"47e453d1-4de1-4c9a-9e56-512e305ab3b4","Type":"ContainerStarted","Data":"b5ab647ccfcb07f6f3f3371ea2f268a7455156890ec645aa2526234b532942e3"} Oct 30 00:58:06.500268 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.500238 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d761e306-1293-4a66-ab30-82658e2ab0fb-kube-api-access-ctzw6" pod "d761e306-1293-4a66-ab30-82658e2ab0fb" (UID: "d761e306-1293-4a66-ab30-82658e2ab0fb"). InnerVolumeSpecName "kube-api-access-ctzw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:06.561453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.537624 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-openshift-ca\") pod \"c64ea606-fc60-41d2-aa91-c7d8352e9988\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " Oct 30 00:58:06.561453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.537774 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64ea606-fc60-41d2-aa91-c7d8352e9988-kube-api-access-hl4fp\" (UniqueName: \"kubernetes.io/projected/c64ea606-fc60-41d2-aa91-c7d8352e9988-kube-api-access-hl4fp\") pod \"c64ea606-fc60-41d2-aa91-c7d8352e9988\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " Oct 30 00:58:06.561453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.537976 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-xks-ca\") pod \"c64ea606-fc60-41d2-aa91-c7d8352e9988\" (UID: \"c64ea606-fc60-41d2-aa91-c7d8352e9988\") " Oct 30 00:58:06.561453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.538769 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctzw6\" (UniqueName: \"kubernetes.io/projected/d761e306-1293-4a66-ab30-82658e2ab0fb-kube-api-access-ctzw6\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:06.561453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.538794 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d761e306-1293-4a66-ab30-82658e2ab0fb-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:06.561453 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.548552 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-openshift-ca" pod "c64ea606-fc60-41d2-aa91-c7d8352e9988" (UID: "c64ea606-fc60-41d2-aa91-c7d8352e9988"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:06.767546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.661118 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:06.907364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.874716 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:06.907364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.881233 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:06.988126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:06.986612 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:07.016050 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.016012 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:58:07.051634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.047363 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:07.051634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.050645 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-xks-ca" pod "c64ea606-fc60-41d2-aa91-c7d8352e9988" (UID: "c64ea606-fc60-41d2-aa91-c7d8352e9988"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:07.051634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.050714 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 00:58:07.051634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.050945 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:07.060578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.052131 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:07.099286 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.079311 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ea606-fc60-41d2-aa91-c7d8352e9988-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:07.106677 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.106638 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 00:58:07.114592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.079393 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:58:07.115083 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.079365 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:58:07.158885 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.081710 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:07.176478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.081743 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:07.391314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.391277 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ea606-fc60-41d2-aa91-c7d8352e9988-kube-api-access-hl4fp" pod "c64ea606-fc60-41d2-aa91-c7d8352e9988" (UID: "c64ea606-fc60-41d2-aa91-c7d8352e9988"). InnerVolumeSpecName "kube-api-access-hl4fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:07.437068 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.402198 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl4fp\" (UniqueName: \"kubernetes.io/projected/c64ea606-fc60-41d2-aa91-c7d8352e9988-kube-api-access-hl4fp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:07.589541 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:58:07.588977 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd761e306_1293_4a66_ab30_82658e2ab0fb.slice/crio-32ae88c94956cfe7e2c9284829826c53a14317c36b9199464dde8f5911d707bd\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:07.786911 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.786881 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 00:58:07.789967 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.789310 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:58:07.789967 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.789460 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:58:07.803702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.803366 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 00:58:07.844699 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.834691 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 00:58:07.844699 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.843122 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq"] Oct 30 00:58:07.844699 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.843170 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" event={"ID":"47e453d1-4de1-4c9a-9e56-512e305ab3b4","Type":"ContainerStarted","Data":"1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496"} Oct 30 00:58:07.844699 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.843201 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:58:07.847235 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.847207 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:58:07.847644 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.847457 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 00:58:07.848660 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.848046 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" event={"ID":"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b","Type":"ContainerStarted","Data":"c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0"} Oct 30 00:58:07.848660 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.848076 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" event={"ID":"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b","Type":"ContainerStarted","Data":"c6b9fa7d5aa341852734d258dedf9ef57783797a4c842d40ab9d1705e37439c9"} Oct 30 00:58:07.848660 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.848423 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz" Oct 30 00:58:07.849273 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.848976 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:58:07.866472 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.866441 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:58:07.868420 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.868269 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 00:58:07.905562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.905535 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 00:58:07.918423 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.918397 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd"] Oct 30 00:58:07.928367 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.928337 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c64ea606-fc60-41d2-aa91-c7d8352e9988" containerName="revoke-keys" Oct 30 00:58:07.928367 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.928364 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ea606-fc60-41d2-aa91-c7d8352e9988" containerName="revoke-keys" Oct 30 00:58:07.928902 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.928875 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d761e306-1293-4a66-ab30-82658e2ab0fb" containerName="revoke-keys" Oct 30 00:58:07.928902 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.928907 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d761e306-1293-4a66-ab30-82658e2ab0fb" containerName="revoke-keys" Oct 30 00:58:07.929335 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.929316 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d761e306-1293-4a66-ab30-82658e2ab0fb" containerName="revoke-keys" Oct 30 00:58:07.929427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.929358 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ea606-fc60-41d2-aa91-c7d8352e9988" containerName="revoke-keys" Oct 30 00:58:07.946868 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.946834 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd"] Oct 30 00:58:07.947022 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.946983 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:07.955097 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.955071 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5"] Oct 30 00:58:07.964267 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.964242 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-baf5781e\"" Oct 30 00:58:07.964586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:07.964566 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-baf5781e-dockercfg-4c5wk\"" Oct 30 00:58:08.115270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.115174 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npzfj\" (UniqueName: \"kubernetes.io/projected/e28fc8a2-0b06-486c-8862-b63f17d1db5d-kube-api-access-npzfj\") pod \"payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:08.115436 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.115391 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fc8a2-0b06-486c-8862-b63f17d1db5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:08.205850 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.205800 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" podStartSLOduration=11.205787666 podStartE2EDuration="11.205787666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:08.200949252 +0000 UTC m=+1173.144122444" watchObservedRunningTime="2026-10-30 00:58:08.205787666 +0000 UTC m=+1173.148960857" Oct 30 00:58:08.216446 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.216418 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npzfj\" (UniqueName: \"kubernetes.io/projected/e28fc8a2-0b06-486c-8862-b63f17d1db5d-kube-api-access-npzfj\") pod \"payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:08.216631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.216594 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fc8a2-0b06-486c-8862-b63f17d1db5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:08.217261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.217236 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fc8a2-0b06-486c-8862-b63f17d1db5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:08.236919 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.236890 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npzfj\" (UniqueName: \"kubernetes.io/projected/e28fc8a2-0b06-486c-8862-b63f17d1db5d-kube-api-access-npzfj\") pod \"payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:08.289350 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.289306 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:08.419146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.419096 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" podStartSLOduration=11.419077108 podStartE2EDuration="11.419077108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:08.416800856 +0000 UTC m=+1173.359974048" watchObservedRunningTime="2026-10-30 00:58:08.419077108 +0000 UTC m=+1173.362250299" Oct 30 00:58:08.679628 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.679536 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd"] Oct 30 00:58:08.701870 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:58:08.701462 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28fc8a2_0b06_486c_8862_b63f17d1db5d.slice/crio-6b12dab5280b000f6d7a9637e67c1081a73337397a57ac7a38e689b926975ebe WatchSource:0}: Error finding container 6b12dab5280b000f6d7a9637e67c1081a73337397a57ac7a38e689b926975ebe: Status 404 returned error can't find the container with id 6b12dab5280b000f6d7a9637e67c1081a73337397a57ac7a38e689b926975ebe Oct 30 00:58:08.859708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.859675 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" event={"ID":"e28fc8a2-0b06-486c-8862-b63f17d1db5d","Type":"ContainerStarted","Data":"aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08"} Oct 30 00:58:08.860163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.859728 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" event={"ID":"e28fc8a2-0b06-486c-8862-b63f17d1db5d","Type":"ContainerStarted","Data":"6b12dab5280b000f6d7a9637e67c1081a73337397a57ac7a38e689b926975ebe"} Oct 30 00:58:08.901314 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:08.901265 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" podStartSLOduration=4.901251544 podStartE2EDuration="4.901251544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:08.900299742 +0000 UTC m=+1173.843472934" watchObservedRunningTime="2026-10-30 00:58:08.901251544 +0000 UTC m=+1173.844424735" Oct 30 00:58:09.871573 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:09.871539 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:12.416507 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:58:12.416474 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:58:15.935747 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:15.934731 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:15.968958 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:15.968320 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:16.064146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.063603 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:16.130562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.130136 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:58:16.408545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.407686 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:16.646544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.645642 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl"] Oct 30 00:58:16.676883 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.676805 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:16.847543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.819945 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:58:16.847543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.820917 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:58:16.847543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.821495 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:58:16.847543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.822076 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:58:17.029603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.982907 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.029603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.983009 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.029603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.983338 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdk9d\" (UniqueName: \"kubernetes.io/projected/9cb02dfe-1464-4f18-9333-b867c4ecd34b-kube-api-access-qdk9d\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.029603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:16.984995 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 00:58:17.083969 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.083918 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:17.116724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.113705 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdk9d\" (UniqueName: \"kubernetes.io/projected/9cb02dfe-1464-4f18-9333-b867c4ecd34b-kube-api-access-qdk9d\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.116724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.113957 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.116724 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.114018 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.139623 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.139583 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:17.184344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.184303 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl"] Oct 30 00:58:17.188241 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.188206 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:58:17.201189 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.201151 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.272062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.272027 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:17.429779 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.398240 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 00:58:17.693600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.683025 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:17.693600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.683088 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.000660 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:17.997735 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.042495 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.042442 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.093424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.093388 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdk9d\" (UniqueName: \"kubernetes.io/projected/9cb02dfe-1464-4f18-9333-b867c4ecd34b-kube-api-access-qdk9d\") pod \"maas-api-revoke-sub-0404966b-dz8gl\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:18.116547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.094225 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.121381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.121340 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.130541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.130486 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.142907 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.142870 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.309823 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.309725 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:18.412182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.411784 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:58:18.412182 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.411975 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.412490 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.412202 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 00:58:18.469674 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.469630 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 00:58:18.471698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.471660 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:18.471846 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.471740 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:18.539745 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:58:18.537628 4905 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.335s" Oct 30 00:58:18.539745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.537793 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:58:18.539745 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.537829 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:58:18.912558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.912129 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:18.912558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:18.912211 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:19.079884 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.079772 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 00:58:19.171643 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.171603 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:19.175627 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.175376 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:19.175627 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.175428 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:19.288096 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.284554 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl"] Oct 30 00:58:19.294263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.292651 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:19.301488 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.301462 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:58:19.387195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.387095 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 00:58:19.413162 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.412911 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:19.415617 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.415059 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:58:19.456547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.456062 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 00:58:19.874237 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.874210 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:58:19.874451 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:19.874283 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:58:20.045360 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:20.045324 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8"] Oct 30 00:58:20.050675 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:20.050649 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53ea6b0-xg5f8"] Oct 30 00:58:20.146270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:20.146092 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" event={"ID":"9cb02dfe-1464-4f18-9333-b867c4ecd34b","Type":"ContainerStarted","Data":"ee6f56dbeb77c66c42debd34b014cbf70b854fa97aec95e06464d93828d6deab"} Oct 30 00:58:20.146270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:20.146189 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" event={"ID":"9cb02dfe-1464-4f18-9333-b867c4ecd34b","Type":"ContainerStarted","Data":"733b8faae5f31e46b018f46d8cbb72b4b760ddf31f70b66b8486c7647f390543"} Oct 30 00:58:20.237871 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:20.237808 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" podStartSLOduration=4.2377884120000004 podStartE2EDuration="4.237788412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:20.194329521 +0000 UTC m=+1185.137502714" watchObservedRunningTime="2026-10-30 00:58:20.237788412 +0000 UTC m=+1185.180961603" Oct 30 00:58:20.883432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:20.883342 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:58:21.165004 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:21.164917 4905 generic.go:356] "Generic (PLEG): container finished" podID="9cb02dfe-1464-4f18-9333-b867c4ecd34b" containerID="ee6f56dbeb77c66c42debd34b014cbf70b854fa97aec95e06464d93828d6deab" exitCode=0 Oct 30 00:58:21.165383 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:21.165009 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" event={"ID":"9cb02dfe-1464-4f18-9333-b867c4ecd34b","Type":"ContainerDied","Data":"ee6f56dbeb77c66c42debd34b014cbf70b854fa97aec95e06464d93828d6deab"} Oct 30 00:58:21.193564 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:21.193536 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b63afef-da01-4e70-8548-be492ba34674" path="/var/lib/kubelet/pods/5b63afef-da01-4e70-8548-be492ba34674/volumes" Oct 30 00:58:22.370474 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.370446 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:22.497408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.497378 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-xks-ca\") pod \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " Oct 30 00:58:22.497629 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.497558 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb02dfe-1464-4f18-9333-b867c4ecd34b-kube-api-access-qdk9d\" (UniqueName: \"kubernetes.io/projected/9cb02dfe-1464-4f18-9333-b867c4ecd34b-kube-api-access-qdk9d\") pod \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " Oct 30 00:58:22.497705 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.497664 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-openshift-ca\") pod \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\" (UID: \"9cb02dfe-1464-4f18-9333-b867c4ecd34b\") " Oct 30 00:58:22.498008 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.497980 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-openshift-ca" pod "9cb02dfe-1464-4f18-9333-b867c4ecd34b" (UID: "9cb02dfe-1464-4f18-9333-b867c4ecd34b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:22.498394 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.498377 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:22.500279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.500255 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-xks-ca" pod "9cb02dfe-1464-4f18-9333-b867c4ecd34b" (UID: "9cb02dfe-1464-4f18-9333-b867c4ecd34b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:22.500365 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.500306 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb02dfe-1464-4f18-9333-b867c4ecd34b-kube-api-access-qdk9d" pod "9cb02dfe-1464-4f18-9333-b867c4ecd34b" (UID: "9cb02dfe-1464-4f18-9333-b867c4ecd34b"). InnerVolumeSpecName "kube-api-access-qdk9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:22.599822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.599787 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb02dfe-1464-4f18-9333-b867c4ecd34b-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:22.599822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:22.599816 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdk9d\" (UniqueName: \"kubernetes.io/projected/9cb02dfe-1464-4f18-9333-b867c4ecd34b-kube-api-access-qdk9d\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:23.249037 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:23.249010 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" Oct 30 00:58:23.259768 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:23.259700 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl" event={"ID":"9cb02dfe-1464-4f18-9333-b867c4ecd34b","Type":"ContainerDied","Data":"733b8faae5f31e46b018f46d8cbb72b4b760ddf31f70b66b8486c7647f390543"} Oct 30 00:58:23.259967 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:23.259816 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733b8faae5f31e46b018f46d8cbb72b4b760ddf31f70b66b8486c7647f390543" Oct 30 00:58:29.532835 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.532802 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9"] Oct 30 00:58:29.552807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.533995 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9cb02dfe-1464-4f18-9333-b867c4ecd34b" containerName="revoke-keys" Oct 30 00:58:29.552807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.534018 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb02dfe-1464-4f18-9333-b867c4ecd34b" containerName="revoke-keys" Oct 30 00:58:29.552807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.534223 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb02dfe-1464-4f18-9333-b867c4ecd34b" containerName="revoke-keys" Oct 30 00:58:29.552807 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.548456 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.571477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.571436 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5f146c27-openshift-default-dockercfg-8sb5r\"" Oct 30 00:58:29.575290 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.575260 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9"] Oct 30 00:58:29.610752 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.610719 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5hj\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-kube-api-access-jv5hj\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.610940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.610847 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-certs\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.610940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.610874 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb00637-ca24-4047-af6b-9535ead35880-istiod-ca-cert\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.610940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.610890 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-data\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.611119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.610961 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-credential-socket\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.611119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.610996 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-socket\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.611119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.611039 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-envoy\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.611119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.611064 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb00637-ca24-4047-af6b-9535ead35880-istio-podinfo\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.611119 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.611105 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-istio-token\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.723545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.717750 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-envoy\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.723878 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.723848 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb00637-ca24-4047-af6b-9535ead35880-istio-podinfo\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.724115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.724095 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-istio-token\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.753144 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.753115 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5hj\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-kube-api-access-jv5hj\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.753667 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.753639 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-certs\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.753877 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.753862 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb00637-ca24-4047-af6b-9535ead35880-istiod-ca-cert\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.754022 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.754005 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-data\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.754943 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.754918 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-data\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.755327 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.755127 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-credential-socket\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.756250 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.755971 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-credential-socket\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.756403 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.756386 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-socket\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.757115 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.757095 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-certs\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.758083 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.758066 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-socket\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.791078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.783600 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb00637-ca24-4047-af6b-9535ead35880-istiod-ca-cert\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.799543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.794114 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5hj\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-kube-api-access-jv5hj\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.799543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.795802 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-envoy\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.824550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.805074 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb00637-ca24-4047-af6b-9535ead35880-istio-podinfo\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.824550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.817993 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-istio-token\") pod \"e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:29.882542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:29.882122 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:30.433747 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:58:30.433685 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb00637_ca24_4047_af6b_9535ead35880.slice/crio-60103cb322bffdba68d1ecd7760d36a6ba4cb094102f461b3a76c57532b4a953 WatchSource:0}: Error finding container 60103cb322bffdba68d1ecd7760d36a6ba4cb094102f461b3a76c57532b4a953: Status 404 returned error can't find the container with id 60103cb322bffdba68d1ecd7760d36a6ba4cb094102f461b3a76c57532b4a953 Oct 30 00:58:30.435021 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:30.434996 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9"] Oct 30 00:58:30.436890 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:30.436846 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:58:30.436992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:30.436932 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:58:30.436992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:30.436977 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:58:31.376895 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:31.376856 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" event={"ID":"9eb00637-ca24-4047-af6b-9535ead35880","Type":"ContainerStarted","Data":"44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927"} Oct 30 00:58:31.377381 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:31.376927 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" event={"ID":"9eb00637-ca24-4047-af6b-9535ead35880","Type":"ContainerStarted","Data":"60103cb322bffdba68d1ecd7760d36a6ba4cb094102f461b3a76c57532b4a953"} Oct 30 00:58:31.441825 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:31.441777 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podStartSLOduration=2.441761459 podStartE2EDuration="2.441761459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:31.437931887 +0000 UTC m=+1196.381105078" watchObservedRunningTime="2026-10-30 00:58:31.441761459 +0000 UTC m=+1196.384934649" Oct 30 00:58:31.882753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:31.882721 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:31.888253 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:31.888228 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:32.419546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:32.418748 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:32.429603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:32.425778 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:58:34.970105 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:34.970070 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs"] Oct 30 00:58:34.975296 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:34.975258 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:34.995201 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:34.995177 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs"] Oct 30 00:58:35.000592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.000572 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5f146c27\"" Oct 30 00:58:35.035308 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.035276 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm"] Oct 30 00:58:35.043060 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.043035 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.068583 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.068264 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm"] Oct 30 00:58:35.142590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.120536 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b5e1c2-af98-46df-9243-c0492304cb15-plugins-config-volume\") pod \"payload-processing-e2e-ait-5f146c27-6885648654-s6lcs\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:35.142590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.120601 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmh9c\" (UniqueName: \"kubernetes.io/projected/a9b5e1c2-af98-46df-9243-c0492304cb15-kube-api-access-qmh9c\") pod \"payload-processing-e2e-ait-5f146c27-6885648654-s6lcs\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:35.142590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.120693 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5gq\" (UniqueName: \"kubernetes.io/projected/5a7f26fb-51ef-4367-88ea-44e43b1fc214-kube-api-access-7z5gq\") pod \"payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.142590 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.120900 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a7f26fb-51ef-4367-88ea-44e43b1fc214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.158717 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.151827 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5f146c27-dockercfg-t5nvq\"" Oct 30 00:58:35.252609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.251894 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b5e1c2-af98-46df-9243-c0492304cb15-plugins-config-volume\") pod \"payload-processing-e2e-ait-5f146c27-6885648654-s6lcs\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:35.252609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.251963 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmh9c\" (UniqueName: \"kubernetes.io/projected/a9b5e1c2-af98-46df-9243-c0492304cb15-kube-api-access-qmh9c\") pod \"payload-processing-e2e-ait-5f146c27-6885648654-s6lcs\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:35.252609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.252071 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5gq\" (UniqueName: \"kubernetes.io/projected/5a7f26fb-51ef-4367-88ea-44e43b1fc214-kube-api-access-7z5gq\") pod \"payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.252609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.252309 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a7f26fb-51ef-4367-88ea-44e43b1fc214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.280540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.272859 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5f146c27\"" Oct 30 00:58:35.284543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.283251 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b5e1c2-af98-46df-9243-c0492304cb15-plugins-config-volume\") pod \"payload-processing-e2e-ait-5f146c27-6885648654-s6lcs\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:35.284543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.284016 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a7f26fb-51ef-4367-88ea-44e43b1fc214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.516367 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.516284 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmh9c\" (UniqueName: \"kubernetes.io/projected/a9b5e1c2-af98-46df-9243-c0492304cb15-kube-api-access-qmh9c\") pod \"payload-processing-e2e-ait-5f146c27-6885648654-s6lcs\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:35.547292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.544223 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5gq\" (UniqueName: \"kubernetes.io/projected/5a7f26fb-51ef-4367-88ea-44e43b1fc214-kube-api-access-7z5gq\") pod \"payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.591494 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.591470 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:58:35.592709 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.592689 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 00:58:35.594718 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.594698 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:35.683415 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.683046 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5f146c27-dockercfg-t5nvq\"" Oct 30 00:58:35.688196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.688168 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:35.903243 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:58:35.903206 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b5e1c2_af98_46df_9243_c0492304cb15.slice/crio-6801339bd5bbde3e8ac00b9329690b1f7a548e4ab1d5edf1ee0c03c95e37381a WatchSource:0}: Error finding container 6801339bd5bbde3e8ac00b9329690b1f7a548e4ab1d5edf1ee0c03c95e37381a: Status 404 returned error can't find the container with id 6801339bd5bbde3e8ac00b9329690b1f7a548e4ab1d5edf1ee0c03c95e37381a Oct 30 00:58:35.903822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:35.903796 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs"] Oct 30 00:58:35.998087 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:58:35.998055 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7f26fb_51ef_4367_88ea_44e43b1fc214.slice/crio-3e10379437a8996963368b1abf5fda08f489875a48367ce73f48f84eaf63e741 WatchSource:0}: Error finding container 3e10379437a8996963368b1abf5fda08f489875a48367ce73f48f84eaf63e741: Status 404 returned error can't find the container with id 3e10379437a8996963368b1abf5fda08f489875a48367ce73f48f84eaf63e741 Oct 30 00:58:36.002839 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.002814 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm"] Oct 30 00:58:36.555428 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.555389 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" event={"ID":"a9b5e1c2-af98-46df-9243-c0492304cb15","Type":"ContainerStarted","Data":"0f5ffa99f5590a0e73f0a5a8e9ff5f0f787029f27cc6c2fb3462c41e9c53295d"} Oct 30 00:58:36.555641 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.555451 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" event={"ID":"a9b5e1c2-af98-46df-9243-c0492304cb15","Type":"ContainerStarted","Data":"6801339bd5bbde3e8ac00b9329690b1f7a548e4ab1d5edf1ee0c03c95e37381a"} Oct 30 00:58:36.555813 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.555785 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:36.557554 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.557507 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" event={"ID":"5a7f26fb-51ef-4367-88ea-44e43b1fc214","Type":"ContainerStarted","Data":"21daf6fbdb87d9a8d7cfbe9b4e8a213837de6654d6809d20acf06f38cfd91f08"} Oct 30 00:58:36.557652 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.557558 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" event={"ID":"5a7f26fb-51ef-4367-88ea-44e43b1fc214","Type":"ContainerStarted","Data":"3e10379437a8996963368b1abf5fda08f489875a48367ce73f48f84eaf63e741"} Oct 30 00:58:36.557728 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.557665 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:36.601407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.601356 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" podStartSLOduration=2.6013436690000002 podStartE2EDuration="2.601343669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:36.598088027 +0000 UTC m=+1201.541261220" watchObservedRunningTime="2026-10-30 00:58:36.601343669 +0000 UTC m=+1201.544516860" Oct 30 00:58:36.639125 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:36.639078 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" podStartSLOduration=1.639066046 podStartE2EDuration="1.639066046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:36.638369559 +0000 UTC m=+1201.581542752" watchObservedRunningTime="2026-10-30 00:58:36.639066046 +0000 UTC m=+1201.582239237" Oct 30 00:58:38.036992 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:38.036955 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct"] Oct 30 00:58:38.063114 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:38.063069 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk"] Oct 30 00:58:38.089740 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:38.089712 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2fc5-qtgct"] Oct 30 00:58:38.097181 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:38.097154 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d9451b-f7pmk"] Oct 30 00:58:39.295545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:39.284726 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2c8a57-acd1-478e-9dcc-abe89c15fbd8" path="/var/lib/kubelet/pods/1f2c8a57-acd1-478e-9dcc-abe89c15fbd8/volumes" Oct 30 00:58:39.295545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:39.285386 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4ea89b-62f6-411e-9130-ede9e25de82c" path="/var/lib/kubelet/pods/2c4ea89b-62f6-411e-9130-ede9e25de82c/volumes" Oct 30 00:58:39.949544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:39.948686 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:39.969544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:39.964273 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:39.969544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:39.964216 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:40.084543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.076985 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 00:58:40.118548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.117881 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:40.194543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.193789 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:58:40.451704 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.451638 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:40.471545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.467593 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:40.749605 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.741915 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p"] Oct 30 00:58:40.821537 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.818863 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:40.846543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.824204 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p"] Oct 30 00:58:40.874091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.862299 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:58:40.874091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.862424 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:58:40.874091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.863192 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:58:40.874091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.865408 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:58:40.909300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.909273 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttks\" (UniqueName: \"kubernetes.io/projected/e47dc303-b0d0-4e2b-9c87-775004e44e36-kube-api-access-8ttks\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:40.909485 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.909458 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:40.909646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:40.909627 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.011180 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.011103 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.011180 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.011176 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.011416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.011366 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttks\" (UniqueName: \"kubernetes.io/projected/e47dc303-b0d0-4e2b-9c87-775004e44e36-kube-api-access-8ttks\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.011764 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.011744 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.013792 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.013770 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.060546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.048985 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttks\" (UniqueName: \"kubernetes.io/projected/e47dc303-b0d0-4e2b-9c87-775004e44e36-kube-api-access-8ttks\") pod \"maas-api-revoke-keys-e2e-derive-a03b2130-58z5p\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.060546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.054096 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6"] Oct 30 00:58:41.078852 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.076886 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f826507-b8kz6"] Oct 30 00:58:41.192343 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.192311 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e64956-5aef-4268-bf87-35637b58ac14" path="/var/lib/kubelet/pods/21e64956-5aef-4268-bf87-35637b58ac14/volumes" Oct 30 00:58:41.207154 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.207124 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:41.566506 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.566315 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p"] Oct 30 00:58:41.823407 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:41.823334 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" event={"ID":"e47dc303-b0d0-4e2b-9c87-775004e44e36","Type":"ContainerStarted","Data":"191ddb2de2caf20875f1432d145dc7516246886eeaa5bf44003d5523176ec21f"} Oct 30 00:58:42.588477 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:58:42.586904 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:58:42.886602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:42.876238 4905 generic.go:356] "Generic (PLEG): container finished" podID="e47dc303-b0d0-4e2b-9c87-775004e44e36" containerID="e7d519cb376e911df829159102d7aaa30d50837bd587796981de57f9bc9ab143" exitCode=0 Oct 30 00:58:42.886602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:42.876617 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" event={"ID":"e47dc303-b0d0-4e2b-9c87-775004e44e36","Type":"ContainerDied","Data":"e7d519cb376e911df829159102d7aaa30d50837bd587796981de57f9bc9ab143"} Oct 30 00:58:44.137578 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.133696 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:44.320130 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.320103 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-openshift-ca\") pod \"e47dc303-b0d0-4e2b-9c87-775004e44e36\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " Oct 30 00:58:44.320427 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.320404 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-openshift-ca" pod "e47dc303-b0d0-4e2b-9c87-775004e44e36" (UID: "e47dc303-b0d0-4e2b-9c87-775004e44e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:44.320534 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.320503 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-xks-ca\") pod \"e47dc303-b0d0-4e2b-9c87-775004e44e36\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " Oct 30 00:58:44.320673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.320655 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47dc303-b0d0-4e2b-9c87-775004e44e36-kube-api-access-8ttks\" (UniqueName: \"kubernetes.io/projected/e47dc303-b0d0-4e2b-9c87-775004e44e36-kube-api-access-8ttks\") pod \"e47dc303-b0d0-4e2b-9c87-775004e44e36\" (UID: \"e47dc303-b0d0-4e2b-9c87-775004e44e36\") " Oct 30 00:58:44.321499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.321476 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:44.322609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.322585 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-xks-ca" pod "e47dc303-b0d0-4e2b-9c87-775004e44e36" (UID: "e47dc303-b0d0-4e2b-9c87-775004e44e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:44.322821 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.322805 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47dc303-b0d0-4e2b-9c87-775004e44e36-kube-api-access-8ttks" pod "e47dc303-b0d0-4e2b-9c87-775004e44e36" (UID: "e47dc303-b0d0-4e2b-9c87-775004e44e36"). InnerVolumeSpecName "kube-api-access-8ttks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:44.422885 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.422820 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ttks\" (UniqueName: \"kubernetes.io/projected/e47dc303-b0d0-4e2b-9c87-775004e44e36-kube-api-access-8ttks\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:44.422885 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.422848 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dc303-b0d0-4e2b-9c87-775004e44e36-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:44.895348 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.895322 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" Oct 30 00:58:44.895530 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.895389 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p" event={"ID":"e47dc303-b0d0-4e2b-9c87-775004e44e36","Type":"ContainerDied","Data":"191ddb2de2caf20875f1432d145dc7516246886eeaa5bf44003d5523176ec21f"} Oct 30 00:58:44.895530 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:44.895441 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="191ddb2de2caf20875f1432d145dc7516246886eeaa5bf44003d5523176ec21f" Oct 30 00:58:47.285090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.285053 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx"] Oct 30 00:58:47.286205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.286187 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e47dc303-b0d0-4e2b-9c87-775004e44e36" containerName="revoke-keys" Oct 30 00:58:47.286205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.286205 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47dc303-b0d0-4e2b-9c87-775004e44e36" containerName="revoke-keys" Oct 30 00:58:47.286399 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.286387 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47dc303-b0d0-4e2b-9c87-775004e44e36" containerName="revoke-keys" Oct 30 00:58:47.290858 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.290828 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.309936 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.309900 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx"] Oct 30 00:58:47.323535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.322234 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:58:47.323535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.322342 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:58:47.323535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.322435 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:58:47.323535 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.322534 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:58:47.324082 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.324045 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.324206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.324152 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.324279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.324254 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9mz\" (UniqueName: \"kubernetes.io/projected/ad382857-c129-4d8f-9d6d-3416643a9d89-kube-api-access-cg9mz\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.425316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.425277 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.425502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.425377 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.425502 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.425460 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9mz\" (UniqueName: \"kubernetes.io/projected/ad382857-c129-4d8f-9d6d-3416643a9d89-kube-api-access-cg9mz\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.426085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.426059 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.427953 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.427931 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.447393 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.447363 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9mz\" (UniqueName: \"kubernetes.io/projected/ad382857-c129-4d8f-9d6d-3416643a9d89-kube-api-access-cg9mz\") pod \"maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:47.570471 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.570397 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:58:47.570471 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.570472 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:58:47.627646 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:47.627350 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:48.004778 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:58:48.004738 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad382857_c129_4d8f_9d6d_3416643a9d89.slice/crio-9874a02b20443257dbcd9695f37c9be607653ef5b2378033b8c53386ba6ad456 WatchSource:0}: Error finding container 9874a02b20443257dbcd9695f37c9be607653ef5b2378033b8c53386ba6ad456: Status 404 returned error can't find the container with id 9874a02b20443257dbcd9695f37c9be607653ef5b2378033b8c53386ba6ad456 Oct 30 00:58:48.007923 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:48.007892 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx"] Oct 30 00:58:48.974753 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:48.974719 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" event={"ID":"ad382857-c129-4d8f-9d6d-3416643a9d89","Type":"ContainerStarted","Data":"5cfa791c345461f8ae4c6178d82303af9df1fb65a9bf7cac6d2dfa8d177f2d7c"} Oct 30 00:58:48.975277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:48.975258 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" event={"ID":"ad382857-c129-4d8f-9d6d-3416643a9d89","Type":"ContainerStarted","Data":"9874a02b20443257dbcd9695f37c9be607653ef5b2378033b8c53386ba6ad456"} Oct 30 00:58:49.019434 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:49.019388 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" podStartSLOduration=2.019374556 podStartE2EDuration="2.019374556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:49.01809069 +0000 UTC m=+1213.961263882" watchObservedRunningTime="2026-10-30 00:58:49.019374556 +0000 UTC m=+1213.962547748" Oct 30 00:58:49.042389 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:49.042335 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg"] Oct 30 00:58:49.048000 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:49.047974 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c700-p2pkg"] Oct 30 00:58:49.194544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:49.194494 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb" path="/var/lib/kubelet/pods/6184d370-9192-4f9c-bbb0-1ee3c8c0c2eb/volumes" Oct 30 00:58:49.988513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:49.988476 4905 generic.go:356] "Generic (PLEG): container finished" podID="ad382857-c129-4d8f-9d6d-3416643a9d89" containerID="5cfa791c345461f8ae4c6178d82303af9df1fb65a9bf7cac6d2dfa8d177f2d7c" exitCode=0 Oct 30 00:58:49.988940 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:49.988569 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" event={"ID":"ad382857-c129-4d8f-9d6d-3416643a9d89","Type":"ContainerDied","Data":"5cfa791c345461f8ae4c6178d82303af9df1fb65a9bf7cac6d2dfa8d177f2d7c"} Oct 30 00:58:51.002607 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.002574 4905 generic.go:356] "Generic (PLEG): container finished" podID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerID="a6ea5984b8ffd61905ac0f0c27ce56f936fa530b26bcc2f81eb91be38e86679e" exitCode=0 Oct 30 00:58:51.003189 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.002656 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" event={"ID":"8e7f16c3-7f67-4664-a08d-2a088130b04b","Type":"ContainerDied","Data":"a6ea5984b8ffd61905ac0f0c27ce56f936fa530b26bcc2f81eb91be38e86679e"} Oct 30 00:58:51.003757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.003287 4905 scope.go:122] "RemoveContainer" containerID="a6ea5984b8ffd61905ac0f0c27ce56f936fa530b26bcc2f81eb91be38e86679e" Oct 30 00:58:51.005288 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.005250 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:58:51.005498 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.005470 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:58:51.005631 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.005551 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:58:51.233597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.232886 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:51.345790 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.345700 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-openshift-ca\") pod \"ad382857-c129-4d8f-9d6d-3416643a9d89\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " Oct 30 00:58:51.345961 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.345840 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-xks-ca\") pod \"ad382857-c129-4d8f-9d6d-3416643a9d89\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " Oct 30 00:58:51.346039 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.345963 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad382857-c129-4d8f-9d6d-3416643a9d89-kube-api-access-cg9mz\" (UniqueName: \"kubernetes.io/projected/ad382857-c129-4d8f-9d6d-3416643a9d89-kube-api-access-cg9mz\") pod \"ad382857-c129-4d8f-9d6d-3416643a9d89\" (UID: \"ad382857-c129-4d8f-9d6d-3416643a9d89\") " Oct 30 00:58:51.346135 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.346107 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-openshift-ca" pod "ad382857-c129-4d8f-9d6d-3416643a9d89" (UID: "ad382857-c129-4d8f-9d6d-3416643a9d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:51.346668 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.346644 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:51.348277 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.348253 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-xks-ca" pod "ad382857-c129-4d8f-9d6d-3416643a9d89" (UID: "ad382857-c129-4d8f-9d6d-3416643a9d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:51.349338 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.349311 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad382857-c129-4d8f-9d6d-3416643a9d89-kube-api-access-cg9mz" pod "ad382857-c129-4d8f-9d6d-3416643a9d89" (UID: "ad382857-c129-4d8f-9d6d-3416643a9d89"). InnerVolumeSpecName "kube-api-access-cg9mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:51.454592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.448846 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad382857-c129-4d8f-9d6d-3416643a9d89-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:51.454592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:51.448887 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg9mz\" (UniqueName: \"kubernetes.io/projected/ad382857-c129-4d8f-9d6d-3416643a9d89-kube-api-access-cg9mz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:52.052331 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.051078 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" event={"ID":"8e7f16c3-7f67-4664-a08d-2a088130b04b","Type":"ContainerStarted","Data":"750979940b14d6a59ab9a8189a2c2d8f7444cccced4b73ae9460cbff0031284a"} Oct 30 00:58:52.102540 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.101656 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj"] Oct 30 00:58:52.171917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.169789 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh"] Oct 30 00:58:52.171917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.171051 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" event={"ID":"ad382857-c129-4d8f-9d6d-3416643a9d89","Type":"ContainerDied","Data":"9874a02b20443257dbcd9695f37c9be607653ef5b2378033b8c53386ba6ad456"} Oct 30 00:58:52.171917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.171151 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9874a02b20443257dbcd9695f37c9be607653ef5b2378033b8c53386ba6ad456" Oct 30 00:58:52.171917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.171260 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx" Oct 30 00:58:52.186403 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.186371 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2485f24b-tqmxj"] Oct 30 00:58:52.193382 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.193358 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dc1479-99xkh"] Oct 30 00:58:52.209896 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.208164 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq"] Oct 30 00:58:52.218704 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.218670 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5"] Oct 30 00:58:52.630550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.629604 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:58:52.630550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.629796 4905 patch_prober.go:29] interesting pod/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:58:52.630550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:52.629864 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 00:58:53.278550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:53.267703 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6d53c3-5657-4926-98c8-033129688a95" path="/var/lib/kubelet/pods/4d6d53c3-5657-4926-98c8-033129688a95/volumes" Oct 30 00:58:53.305794 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:53.305762 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0480cf-7b60-4aa2-9536-5f541ad3aa1e" path="/var/lib/kubelet/pods/6a0480cf-7b60-4aa2-9536-5f541ad3aa1e/volumes" Oct 30 00:58:53.627386 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:53.627304 4905 patch_prober.go:29] interesting pod/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:58:53.627386 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:53.627360 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 00:58:54.636544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:54.624739 4905 patch_prober.go:29] interesting pod/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:58:54.636544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:54.624795 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 00:58:55.043114 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:55.043084 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:58:55.624179 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:55.624144 4905 patch_prober.go:29] interesting pod/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:58:55.624352 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:55.624195 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 00:58:56.635545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:56.629068 4905 patch_prober.go:29] interesting pod/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:58:56.635545 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:56.629187 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 00:58:57.208822 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.208783 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" podUID="47e453d1-4de1-4c9a-9e56-512e305ab3b4" containerName="payload-processing" containerID="cri-o://1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496" gracePeriod=25 Oct 30 00:58:57.219404 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.219358 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" podUID="f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" containerName="payload-pre-processing" containerID="cri-o://c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0" gracePeriod=25 Oct 30 00:58:57.569968 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.569939 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:58:57.617983 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.617948 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:58:57.693892 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.693841 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:58:57.695124 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.695074 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e453d1-4de1-4c9a-9e56-512e305ab3b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e453d1-4de1-4c9a-9e56-512e305ab3b4-plugins-config-volume\") pod \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " Oct 30 00:58:57.695233 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.695186 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-kube-api-access-pxpsb\" (UniqueName: \"kubernetes.io/projected/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-kube-api-access-pxpsb\") pod \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " Oct 30 00:58:57.695416 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.695394 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e453d1-4de1-4c9a-9e56-512e305ab3b4-kube-api-access-xq8pn\" (UniqueName: \"kubernetes.io/projected/47e453d1-4de1-4c9a-9e56-512e305ab3b4-kube-api-access-xq8pn\") pod \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\" (UID: \"47e453d1-4de1-4c9a-9e56-512e305ab3b4\") " Oct 30 00:58:57.695501 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.695431 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-plugins-config-volume\") pod \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\" (UID: \"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b\") " Oct 30 00:58:57.696571 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.696541 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-plugins-config-volume" pod "f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" (UID: "f82a0bba-52c6-4e01-9fdc-fd5369b29e8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:57.696957 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.696936 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:58:57.697492 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.697463 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e453d1-4de1-4c9a-9e56-512e305ab3b4-plugins-config-volume" pod "47e453d1-4de1-4c9a-9e56-512e305ab3b4" (UID: "47e453d1-4de1-4c9a-9e56-512e305ab3b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:57.699098 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.699068 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-kube-api-access-pxpsb" pod "f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" (UID: "f82a0bba-52c6-4e01-9fdc-fd5369b29e8b"). InnerVolumeSpecName "kube-api-access-pxpsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:57.699313 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.699290 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e453d1-4de1-4c9a-9e56-512e305ab3b4-kube-api-access-xq8pn" pod "47e453d1-4de1-4c9a-9e56-512e305ab3b4" (UID: "47e453d1-4de1-4c9a-9e56-512e305ab3b4"). InnerVolumeSpecName "kube-api-access-xq8pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:57.800257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.798231 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e453d1-4de1-4c9a-9e56-512e305ab3b4-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:57.800257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.798264 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxpsb\" (UniqueName: \"kubernetes.io/projected/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-kube-api-access-pxpsb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:57.800257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.798280 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq8pn\" (UniqueName: \"kubernetes.io/projected/47e453d1-4de1-4c9a-9e56-512e305ab3b4-kube-api-access-xq8pn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:57.800257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.798293 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:58:57.842491 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:57.842454 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd"] Oct 30 00:58:58.675595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.657249 4905 generic.go:356] "Generic (PLEG): container finished" podID="47e453d1-4de1-4c9a-9e56-512e305ab3b4" containerID="1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496" exitCode=0 Oct 30 00:58:58.675595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.663054 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" event={"ID":"47e453d1-4de1-4c9a-9e56-512e305ab3b4","Type":"ContainerDied","Data":"1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496"} Oct 30 00:58:58.675595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.663194 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" event={"ID":"47e453d1-4de1-4c9a-9e56-512e305ab3b4","Type":"ContainerDied","Data":"b5ab647ccfcb07f6f3f3371ea2f268a7455156890ec645aa2526234b532942e3"} Oct 30 00:58:58.675595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.663242 4905 scope.go:122] "RemoveContainer" containerID="1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496" Oct 30 00:58:58.675595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.663445 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq" Oct 30 00:58:58.694838 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.690774 4905 generic.go:356] "Generic (PLEG): container finished" podID="f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" containerID="c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0" exitCode=0 Oct 30 00:58:58.697977 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.697903 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" Oct 30 00:58:58.706997 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.706960 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" event={"ID":"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b","Type":"ContainerDied","Data":"c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0"} Oct 30 00:58:58.707146 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.707030 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5" event={"ID":"f82a0bba-52c6-4e01-9fdc-fd5369b29e8b","Type":"ContainerDied","Data":"c6b9fa7d5aa341852734d258dedf9ef57783797a4c842d40ab9d1705e37439c9"} Oct 30 00:58:58.741684 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.741654 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p"] Oct 30 00:58:58.970006 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:58.967567 4905 scope.go:122] "RemoveContainer" containerID="1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496" Oct 30 00:58:59.059587 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:58:59.035923 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496\": container with ID starting with 1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496 not found: ID does not exist" containerID="1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496" Oct 30 00:58:59.059587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.035979 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496"} err="failed to get container status \"1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496\": rpc error: code = NotFound desc = could not find container \"1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496\": container with ID starting with 1e06a815d9e1bb7e5856399c19e997f9039674b9f14e5c0ba8acac2336658496 not found: ID does not exist" Oct 30 00:58:59.059587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.036011 4905 scope.go:122] "RemoveContainer" containerID="c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0" Oct 30 00:58:59.333546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.303930 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:59.541214 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.482598 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:59.694600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.679830 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a03b2130-58z5p"] Oct 30 00:58:59.694857 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.694834 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l"] Oct 30 00:58:59.777592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.775541 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd373ed-tkl6l"] Oct 30 00:58:59.792548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.791257 4905 scope.go:122] "RemoveContainer" containerID="c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0" Oct 30 00:58:59.795550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.792898 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5"] Oct 30 00:58:59.795550 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:58:59.792994 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0\": container with ID starting with c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0 not found: ID does not exist" containerID="c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0" Oct 30 00:58:59.795550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.793045 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0"} err="failed to get container status \"c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0\": rpc error: code = NotFound desc = could not find container \"c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0\": container with ID starting with c7707dbe229cff9fcf5b4ac971e20881638057f64d6726a4540b5e58021df7a0 not found: ID does not exist" Oct 30 00:58:59.807592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.801622 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a03b2130-6d7bd9d8c5-w6gt5"] Oct 30 00:58:59.807803 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.807781 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq"] Oct 30 00:58:59.815587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:58:59.813595 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a03b2130-5fdb85bffb-rpgmq"] Oct 30 00:59:01.031767 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.031723 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7"] Oct 30 00:59:01.045194 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.045164 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w"] Oct 30 00:59:01.050915 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.050888 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be7f7d05-6b7s7"] Oct 30 00:59:01.056177 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.056155 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d011ba-5qp4w"] Oct 30 00:59:01.192484 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.192448 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e453d1-4de1-4c9a-9e56-512e305ab3b4" path="/var/lib/kubelet/pods/47e453d1-4de1-4c9a-9e56-512e305ab3b4/volumes" Oct 30 00:59:01.192989 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.192973 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c" path="/var/lib/kubelet/pods/8f08ddaf-ce67-44a7-8d8d-51e95e0e5c0c/volumes" Oct 30 00:59:01.193385 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.193372 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9df9242-a84a-4fa1-b23d-d86b7a4f209e" path="/var/lib/kubelet/pods/c9df9242-a84a-4fa1-b23d-d86b7a4f209e/volumes" Oct 30 00:59:01.194125 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.194109 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47dc303-b0d0-4e2b-9c87-775004e44e36" path="/var/lib/kubelet/pods/e47dc303-b0d0-4e2b-9c87-775004e44e36/volumes" Oct 30 00:59:01.194576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.194561 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ea385e-402b-4ba5-8434-2e7cdb47c3f1" path="/var/lib/kubelet/pods/e5ea385e-402b-4ba5-8434-2e7cdb47c3f1/volumes" Oct 30 00:59:01.194929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.194917 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" path="/var/lib/kubelet/pods/f82a0bba-52c6-4e01-9fdc-fd5369b29e8b/volumes" Oct 30 00:59:01.565074 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565038 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w"] Oct 30 00:59:01.565751 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565733 4905 cpu_manager.go:414] "containerMap: removing container" podUID="47e453d1-4de1-4c9a-9e56-512e305ab3b4" containerName="payload-processing" Oct 30 00:59:01.565751 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565750 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e453d1-4de1-4c9a-9e56-512e305ab3b4" containerName="payload-processing" Oct 30 00:59:01.565929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565793 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ad382857-c129-4d8f-9d6d-3416643a9d89" containerName="revoke-keys" Oct 30 00:59:01.565929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565799 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad382857-c129-4d8f-9d6d-3416643a9d89" containerName="revoke-keys" Oct 30 00:59:01.565929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565809 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" containerName="payload-pre-processing" Oct 30 00:59:01.565929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565814 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" containerName="payload-pre-processing" Oct 30 00:59:01.565929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565921 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad382857-c129-4d8f-9d6d-3416643a9d89" containerName="revoke-keys" Oct 30 00:59:01.565929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565931 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82a0bba-52c6-4e01-9fdc-fd5369b29e8b" containerName="payload-pre-processing" Oct 30 00:59:01.566228 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.565941 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e453d1-4de1-4c9a-9e56-512e305ab3b4" containerName="payload-processing" Oct 30 00:59:01.583812 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.583781 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.590624 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.590600 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w"] Oct 30 00:59:01.600258 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.600233 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:59:01.613881 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.613861 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:59:01.613997 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.613958 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:59:01.614099 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.614085 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:59:01.664499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.664454 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.664697 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.664512 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.664814 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.664791 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf45m\" (UniqueName: \"kubernetes.io/projected/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-kube-api-access-lf45m\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.766592 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.766560 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf45m\" (UniqueName: \"kubernetes.io/projected/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-kube-api-access-lf45m\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.766781 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.766690 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.766781 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.766730 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.767329 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.767308 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.769305 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.769276 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.787027 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.787005 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf45m\" (UniqueName: \"kubernetes.io/projected/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-kube-api-access-lf45m\") pod \"maas-api-revoke-sub-ac95ba23-sfd8w\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:01.901301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:01.901213 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:02.212473 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:59:02.212445 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ac3fef_1c9e_47ae_8a2f_e5e67105596c.slice/crio-cd27594f3b13d995ffefbb1290d421e89d7255c2819c0d53a0c6164f73784deb WatchSource:0}: Error finding container cd27594f3b13d995ffefbb1290d421e89d7255c2819c0d53a0c6164f73784deb: Status 404 returned error can't find the container with id cd27594f3b13d995ffefbb1290d421e89d7255c2819c0d53a0c6164f73784deb Oct 30 00:59:02.216270 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:02.216247 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w"] Oct 30 00:59:02.843958 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:02.843861 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" podUID="e28fc8a2-0b06-486c-8862-b63f17d1db5d" containerName="payload-pre-processing" containerID="cri-o://aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08" gracePeriod=25 Oct 30 00:59:02.857054 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:02.857015 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" event={"ID":"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c","Type":"ContainerStarted","Data":"fa56d903ced590e176179f871fbc079ed501d87336e4e46ac1eb96d2a270e76c"} Oct 30 00:59:02.857192 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:02.857087 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" event={"ID":"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c","Type":"ContainerStarted","Data":"cd27594f3b13d995ffefbb1290d421e89d7255c2819c0d53a0c6164f73784deb"} Oct 30 00:59:02.904634 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:02.904565 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" podStartSLOduration=1.904549744 podStartE2EDuration="1.904549744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:02.898501517 +0000 UTC m=+1227.841674709" watchObservedRunningTime="2026-10-30 00:59:02.904549744 +0000 UTC m=+1227.847722940" Oct 30 00:59:03.207454 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.207424 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:59:03.306046 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.306015 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28fc8a2-0b06-486c-8862-b63f17d1db5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fc8a2-0b06-486c-8862-b63f17d1db5d-plugins-config-volume\") pod \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " Oct 30 00:59:03.306542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.306087 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28fc8a2-0b06-486c-8862-b63f17d1db5d-kube-api-access-npzfj\" (UniqueName: \"kubernetes.io/projected/e28fc8a2-0b06-486c-8862-b63f17d1db5d-kube-api-access-npzfj\") pod \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\" (UID: \"e28fc8a2-0b06-486c-8862-b63f17d1db5d\") " Oct 30 00:59:03.306542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.306363 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28fc8a2-0b06-486c-8862-b63f17d1db5d-plugins-config-volume" pod "e28fc8a2-0b06-486c-8862-b63f17d1db5d" (UID: "e28fc8a2-0b06-486c-8862-b63f17d1db5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:03.306671 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.306569 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fc8a2-0b06-486c-8862-b63f17d1db5d-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.308261 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.308242 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28fc8a2-0b06-486c-8862-b63f17d1db5d-kube-api-access-npzfj" pod "e28fc8a2-0b06-486c-8862-b63f17d1db5d" (UID: "e28fc8a2-0b06-486c-8862-b63f17d1db5d"). InnerVolumeSpecName "kube-api-access-npzfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:03.407628 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.407556 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npzfj\" (UniqueName: \"kubernetes.io/projected/e28fc8a2-0b06-486c-8862-b63f17d1db5d-kube-api-access-npzfj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.874195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.874163 4905 generic.go:356] "Generic (PLEG): container finished" podID="d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" containerID="fa56d903ced590e176179f871fbc079ed501d87336e4e46ac1eb96d2a270e76c" exitCode=0 Oct 30 00:59:03.874449 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.874316 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" event={"ID":"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c","Type":"ContainerDied","Data":"fa56d903ced590e176179f871fbc079ed501d87336e4e46ac1eb96d2a270e76c"} Oct 30 00:59:03.876085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.876045 4905 generic.go:356] "Generic (PLEG): container finished" podID="e28fc8a2-0b06-486c-8862-b63f17d1db5d" containerID="aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08" exitCode=0 Oct 30 00:59:03.876085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.876074 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" event={"ID":"e28fc8a2-0b06-486c-8862-b63f17d1db5d","Type":"ContainerDied","Data":"aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08"} Oct 30 00:59:03.876085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.876088 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" Oct 30 00:59:03.876344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.876113 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd" event={"ID":"e28fc8a2-0b06-486c-8862-b63f17d1db5d","Type":"ContainerDied","Data":"6b12dab5280b000f6d7a9637e67c1081a73337397a57ac7a38e689b926975ebe"} Oct 30 00:59:03.876344 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.876140 4905 scope.go:122] "RemoveContainer" containerID="aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08" Oct 30 00:59:03.887629 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.887608 4905 scope.go:122] "RemoveContainer" containerID="aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08" Oct 30 00:59:03.887886 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:59:03.887869 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08\": container with ID starting with aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08 not found: ID does not exist" containerID="aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08" Oct 30 00:59:03.887930 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.887895 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08"} err="failed to get container status \"aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08\": rpc error: code = NotFound desc = could not find container \"aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08\": container with ID starting with aa7ff55b628be3883ee2790883ac3503141b206dbb61e5946ff241eb096b2a08 not found: ID does not exist" Oct 30 00:59:03.960429 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.960371 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd"] Oct 30 00:59:03.966054 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:03.966027 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-baf5781e-58cb4779dc-d5vvd"] Oct 30 00:59:05.073098 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.073071 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:05.187929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.180482 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-kube-api-access-lf45m\" (UniqueName: \"kubernetes.io/projected/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-kube-api-access-lf45m\") pod \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " Oct 30 00:59:05.187929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.180632 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-openshift-ca\") pod \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " Oct 30 00:59:05.187929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.180685 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-xks-ca\") pod \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\" (UID: \"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c\") " Oct 30 00:59:05.219551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.215920 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-openshift-ca" pod "d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" (UID: "d4ac3fef-1c9e-47ae-8a2f-e5e67105596c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:05.249403 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.249361 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-xks-ca" pod "d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" (UID: "d4ac3fef-1c9e-47ae-8a2f-e5e67105596c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:05.252128 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.252094 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28fc8a2-0b06-486c-8862-b63f17d1db5d" path="/var/lib/kubelet/pods/e28fc8a2-0b06-486c-8862-b63f17d1db5d/volumes" Oct 30 00:59:05.255063 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.255037 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-kube-api-access-lf45m" pod "d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" (UID: "d4ac3fef-1c9e-47ae-8a2f-e5e67105596c"). InnerVolumeSpecName "kube-api-access-lf45m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:05.282776 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.282747 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf45m\" (UniqueName: \"kubernetes.io/projected/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-kube-api-access-lf45m\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:05.282888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.282781 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:05.282888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.282796 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:05.922553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.922181 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" event={"ID":"d4ac3fef-1c9e-47ae-8a2f-e5e67105596c","Type":"ContainerDied","Data":"cd27594f3b13d995ffefbb1290d421e89d7255c2819c0d53a0c6164f73784deb"} Oct 30 00:59:05.922553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.922230 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd27594f3b13d995ffefbb1290d421e89d7255c2819c0d53a0c6164f73784deb" Oct 30 00:59:05.922553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:05.922289 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w" Oct 30 00:59:06.040937 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:06.032527 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk"] Oct 30 00:59:06.048587 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:06.041944 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7f48a62-znvdk"] Oct 30 00:59:07.267717 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.267677 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ea931d-3fd7-4c23-994b-902b8cedfe2a" path="/var/lib/kubelet/pods/e3ea931d-3fd7-4c23-994b-902b8cedfe2a/volumes" Oct 30 00:59:07.268835 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.268810 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9"] Oct 30 00:59:07.270079 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.270046 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" containerName="revoke-keys" Oct 30 00:59:07.270334 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.270291 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" containerName="revoke-keys" Oct 30 00:59:07.270762 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.270745 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e28fc8a2-0b06-486c-8862-b63f17d1db5d" containerName="payload-pre-processing" Oct 30 00:59:07.270879 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.270866 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28fc8a2-0b06-486c-8862-b63f17d1db5d" containerName="payload-pre-processing" Oct 30 00:59:07.271215 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.271188 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" containerName="revoke-keys" Oct 30 00:59:07.271346 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.271334 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28fc8a2-0b06-486c-8862-b63f17d1db5d" containerName="payload-pre-processing" Oct 30 00:59:07.277551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.277511 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9"] Oct 30 00:59:07.277802 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.277723 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.296513 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.296493 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:59:07.311782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.311759 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:59:07.311895 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.311880 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:59:07.311939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.311930 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:59:07.379788 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.379757 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gff\" (UniqueName: \"kubernetes.io/projected/9f530630-85af-484a-9451-d9d82052f994-kube-api-access-c5gff\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.379918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.379820 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f530630-85af-484a-9451-d9d82052f994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.379918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.379884 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f530630-85af-484a-9451-d9d82052f994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.481090 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.481064 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gff\" (UniqueName: \"kubernetes.io/projected/9f530630-85af-484a-9451-d9d82052f994-kube-api-access-c5gff\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.481250 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.481127 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f530630-85af-484a-9451-d9d82052f994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.481250 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.481183 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f530630-85af-484a-9451-d9d82052f994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.482004 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.481980 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f530630-85af-484a-9451-d9d82052f994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.483839 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.483816 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f530630-85af-484a-9451-d9d82052f994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.502067 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.502041 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gff\" (UniqueName: \"kubernetes.io/projected/9f530630-85af-484a-9451-d9d82052f994-kube-api-access-c5gff\") pod \"maas-api-revoke-sub-6e0997bc-7vkl9\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.598923 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.598845 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:07.975387 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:07.966244 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9"] Oct 30 00:59:09.006424 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:09.006391 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" event={"ID":"9f530630-85af-484a-9451-d9d82052f994","Type":"ContainerStarted","Data":"da4eed6ce66ada5f1eff37b8cbf28eaa671f8d8cabd3c1020c7526585458a782"} Oct 30 00:59:09.006815 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:09.006443 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" event={"ID":"9f530630-85af-484a-9451-d9d82052f994","Type":"ContainerStarted","Data":"efd3dcbf52f6cd206885ca22bbfc7eb733542d5b6e2da00fa3439c5cf4412b60"} Oct 30 00:59:09.060913 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:09.059189 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" podStartSLOduration=3.059172431 podStartE2EDuration="3.059172431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:09.057254311 +0000 UTC m=+1234.000427504" watchObservedRunningTime="2026-10-30 00:59:09.059172431 +0000 UTC m=+1234.002345624" Oct 30 00:59:10.016977 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:10.016945 4905 generic.go:356] "Generic (PLEG): container finished" podID="9f530630-85af-484a-9451-d9d82052f994" containerID="da4eed6ce66ada5f1eff37b8cbf28eaa671f8d8cabd3c1020c7526585458a782" exitCode=0 Oct 30 00:59:10.017363 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:10.017044 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" event={"ID":"9f530630-85af-484a-9451-d9d82052f994","Type":"ContainerDied","Data":"da4eed6ce66ada5f1eff37b8cbf28eaa671f8d8cabd3c1020c7526585458a782"} Oct 30 00:59:11.390432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.390402 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:11.501295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.501257 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f530630-85af-484a-9451-d9d82052f994-kube-api-access-c5gff\" (UniqueName: \"kubernetes.io/projected/9f530630-85af-484a-9451-d9d82052f994-kube-api-access-c5gff\") pod \"9f530630-85af-484a-9451-d9d82052f994\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " Oct 30 00:59:11.501466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.501407 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f530630-85af-484a-9451-d9d82052f994-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f530630-85af-484a-9451-d9d82052f994-maas-api-xks-ca\") pod \"9f530630-85af-484a-9451-d9d82052f994\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " Oct 30 00:59:11.501466 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.501461 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f530630-85af-484a-9451-d9d82052f994-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f530630-85af-484a-9451-d9d82052f994-maas-api-openshift-ca\") pod \"9f530630-85af-484a-9451-d9d82052f994\" (UID: \"9f530630-85af-484a-9451-d9d82052f994\") " Oct 30 00:59:11.501975 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.501948 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f530630-85af-484a-9451-d9d82052f994-maas-api-openshift-ca" pod "9f530630-85af-484a-9451-d9d82052f994" (UID: "9f530630-85af-484a-9451-d9d82052f994"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:11.502132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.502114 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f530630-85af-484a-9451-d9d82052f994-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:11.504452 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.504426 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f530630-85af-484a-9451-d9d82052f994-maas-api-xks-ca" pod "9f530630-85af-484a-9451-d9d82052f994" (UID: "9f530630-85af-484a-9451-d9d82052f994"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:11.505078 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.505059 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f530630-85af-484a-9451-d9d82052f994-kube-api-access-c5gff" pod "9f530630-85af-484a-9451-d9d82052f994" (UID: "9f530630-85af-484a-9451-d9d82052f994"). InnerVolumeSpecName "kube-api-access-c5gff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:11.604244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.604162 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5gff\" (UniqueName: \"kubernetes.io/projected/9f530630-85af-484a-9451-d9d82052f994-kube-api-access-c5gff\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:11.604244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:11.604188 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f530630-85af-484a-9451-d9d82052f994-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:12.074771 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:12.074739 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" event={"ID":"9f530630-85af-484a-9451-d9d82052f994","Type":"ContainerDied","Data":"efd3dcbf52f6cd206885ca22bbfc7eb733542d5b6e2da00fa3439c5cf4412b60"} Oct 30 00:59:12.074969 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:12.074795 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efd3dcbf52f6cd206885ca22bbfc7eb733542d5b6e2da00fa3439c5cf4412b60" Oct 30 00:59:12.074969 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:12.074851 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9" Oct 30 00:59:12.414989 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:59:12.414910 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:59:14.643701 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.643664 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr"] Oct 30 00:59:14.647364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.644820 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9f530630-85af-484a-9451-d9d82052f994" containerName="revoke-keys" Oct 30 00:59:14.647364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.644844 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f530630-85af-484a-9451-d9d82052f994" containerName="revoke-keys" Oct 30 00:59:14.647364 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.645054 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f530630-85af-484a-9451-d9d82052f994" containerName="revoke-keys" Oct 30 00:59:14.650016 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.649992 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.663766 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.662299 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr"] Oct 30 00:59:14.669091 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.669005 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:59:14.669797 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.669775 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:59:14.669938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.669793 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:59:14.680266 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.680244 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:59:14.756357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.756314 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m658j\" (UniqueName: \"kubernetes.io/projected/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-kube-api-access-m658j\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.756686 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.756663 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.756926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.756896 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.858354 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.858309 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.858558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.858425 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.858558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.858550 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m658j\" (UniqueName: \"kubernetes.io/projected/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-kube-api-access-m658j\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.859065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.859036 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.861019 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.860996 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.879112 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.879091 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m658j\" (UniqueName: \"kubernetes.io/projected/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-kube-api-access-m658j\") pod \"maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:14.972656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:14.972580 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:15.288277 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:59:15.288247 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ccd9eb_7c63_4a34_8bfc_d6db54926cda.slice/crio-44cc5d74d3f3657118655e5fd3cc2e47a5fc771d51efe6aaaa2bc23a120dd4e1 WatchSource:0}: Error finding container 44cc5d74d3f3657118655e5fd3cc2e47a5fc771d51efe6aaaa2bc23a120dd4e1: Status 404 returned error can't find the container with id 44cc5d74d3f3657118655e5fd3cc2e47a5fc771d51efe6aaaa2bc23a120dd4e1 Oct 30 00:59:15.292132 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:15.292109 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr"] Oct 30 00:59:16.123449 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.123376 4905 generic.go:356] "Generic (PLEG): container finished" podID="9eb00637-ca24-4047-af6b-9535ead35880" containerID="44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927" exitCode=0 Oct 30 00:59:16.123888 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.123451 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" event={"ID":"9eb00637-ca24-4047-af6b-9535ead35880","Type":"ContainerDied","Data":"44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927"} Oct 30 00:59:16.124011 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.123991 4905 scope.go:122] "RemoveContainer" containerID="44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927" Oct 30 00:59:16.125698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.125665 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:59:16.125828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.125752 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:59:16.125828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.125804 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:59:16.125943 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.125859 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" event={"ID":"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda","Type":"ContainerStarted","Data":"ea48301d88dc997ec6c1707d73d013e15bc5586b6bffbbee4de92d3ad0c6eaa2"} Oct 30 00:59:16.125943 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.125895 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" event={"ID":"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda","Type":"ContainerStarted","Data":"44cc5d74d3f3657118655e5fd3cc2e47a5fc771d51efe6aaaa2bc23a120dd4e1"} Oct 30 00:59:16.224429 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.224371 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" podStartSLOduration=2.224353054 podStartE2EDuration="2.224353054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:16.222945429 +0000 UTC m=+1241.166118622" watchObservedRunningTime="2026-10-30 00:59:16.224353054 +0000 UTC m=+1241.167526248" Oct 30 00:59:16.484842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.484813 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx"] Oct 30 00:59:16.490167 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.490143 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-baf5781e-d8zrx"] Oct 30 00:59:16.683835 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.683799 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62"] Oct 30 00:59:16.684285 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:16.684255 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" containerID="cri-o://750979940b14d6a59ab9a8189a2c2d8f7444cccced4b73ae9460cbff0031284a" gracePeriod=30 Oct 30 00:59:17.014553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.007536 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5"] Oct 30 00:59:17.309196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.287005 4905 generic.go:356] "Generic (PLEG): container finished" podID="e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" containerID="ea48301d88dc997ec6c1707d73d013e15bc5586b6bffbbee4de92d3ad0c6eaa2" exitCode=0 Oct 30 00:59:17.309196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.303790 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" containerID="cri-o://c27235258f8a83d4327f15f57afe2133fd5850c7e584c5cdabef56781cc24d9a" gracePeriod=30 Oct 30 00:59:17.321152 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.321123 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad382857-c129-4d8f-9d6d-3416643a9d89" path="/var/lib/kubelet/pods/ad382857-c129-4d8f-9d6d-3416643a9d89/volumes" Oct 30 00:59:17.321625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.321606 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz"] Oct 30 00:59:17.321708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.321641 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh"] Oct 30 00:59:17.321708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.321661 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1d4f71-qjtzz"] Oct 30 00:59:17.321708 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.321684 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" event={"ID":"9eb00637-ca24-4047-af6b-9535ead35880","Type":"ContainerStarted","Data":"985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186"} Oct 30 00:59:17.321812 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.321719 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" event={"ID":"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda","Type":"ContainerDied","Data":"ea48301d88dc997ec6c1707d73d013e15bc5586b6bffbbee4de92d3ad0c6eaa2"} Oct 30 00:59:17.359729 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.359692 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2fd1-7bvhh"] Oct 30 00:59:17.419221 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.419192 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:59:17.883607 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.883578 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:59:17.884420 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.884400 4905 patch_prober.go:29] interesting pod/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:59:17.884470 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:17.884444 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:59:18.362552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:18.361008 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:59:18.362552 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:18.361073 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:59:18.890120 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:18.890087 4905 patch_prober.go:29] interesting pod/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:59:18.904939 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:18.904888 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:59:19.117568 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.117545 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:19.328614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.314097 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-xks-ca\") pod \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " Oct 30 00:59:19.328614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.314234 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-openshift-ca\") pod \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " Oct 30 00:59:19.328614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.314625 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-kube-api-access-m658j\" (UniqueName: \"kubernetes.io/projected/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-kube-api-access-m658j\") pod \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\" (UID: \"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda\") " Oct 30 00:59:19.328614 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.320896 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-openshift-ca" pod "e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" (UID: "e5ccd9eb-7c63-4a34-8bfc-d6db54926cda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:19.447548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.422493 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:19.447548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.425551 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707e1469-699c-4fc5-8975-53d4e5392334" path="/var/lib/kubelet/pods/707e1469-699c-4fc5-8975-53d4e5392334/volumes" Oct 30 00:59:19.447548 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.426175 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8262f00-9ea2-4828-acb7-707c32b9f355" path="/var/lib/kubelet/pods/a8262f00-9ea2-4828-acb7-707c32b9f355/volumes" Oct 30 00:59:19.474541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.474143 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-xks-ca" pod "e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" (UID: "e5ccd9eb-7c63-4a34-8bfc-d6db54926cda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:19.475673 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.475630 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-kube-api-access-m658j" pod "e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" (UID: "e5ccd9eb-7c63-4a34-8bfc-d6db54926cda"). InnerVolumeSpecName "kube-api-access-m658j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:19.476126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.476103 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" Oct 30 00:59:19.476327 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.476292 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr" event={"ID":"e5ccd9eb-7c63-4a34-8bfc-d6db54926cda","Type":"ContainerDied","Data":"44cc5d74d3f3657118655e5fd3cc2e47a5fc771d51efe6aaaa2bc23a120dd4e1"} Oct 30 00:59:19.476478 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.476458 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44cc5d74d3f3657118655e5fd3cc2e47a5fc771d51efe6aaaa2bc23a120dd4e1" Oct 30 00:59:19.524455 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.524434 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m658j\" (UniqueName: \"kubernetes.io/projected/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-kube-api-access-m658j\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:19.524613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.524458 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:19.884603 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.884567 4905 patch_prober.go:29] interesting pod/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:59:19.884820 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:19.884650 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:59:20.883562 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:20.883491 4905 patch_prober.go:29] interesting pod/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:59:20.884026 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:20.883579 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:59:21.883263 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:21.883227 4905 patch_prober.go:29] interesting pod/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:59:21.883435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:21.883282 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:59:22.523786 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.523740 4905 generic.go:356] "Generic (PLEG): container finished" podID="6392db76-3331-49b5-9bf3-055e24d0f203" containerID="c27235258f8a83d4327f15f57afe2133fd5850c7e584c5cdabef56781cc24d9a" exitCode=0 Oct 30 00:59:22.524150 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.523813 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" event={"ID":"6392db76-3331-49b5-9bf3-055e24d0f203","Type":"ContainerDied","Data":"c27235258f8a83d4327f15f57afe2133fd5850c7e584c5cdabef56781cc24d9a"} Oct 30 00:59:22.524150 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.523924 4905 scope.go:122] "RemoveContainer" containerID="04909fc8e66b995e3b0d85b10ff8e024e884703509bf2b86275a93b164d631df" Oct 30 00:59:22.563804 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.563769 4905 generic.go:356] "Generic (PLEG): container finished" podID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerID="750979940b14d6a59ab9a8189a2c2d8f7444cccced4b73ae9460cbff0031284a" exitCode=0 Oct 30 00:59:22.564077 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.564057 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" event={"ID":"8e7f16c3-7f67-4664-a08d-2a088130b04b","Type":"ContainerDied","Data":"750979940b14d6a59ab9a8189a2c2d8f7444cccced4b73ae9460cbff0031284a"} Oct 30 00:59:22.580756 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.580732 4905 scope.go:122] "RemoveContainer" containerID="a6ea5984b8ffd61905ac0f0c27ce56f936fa530b26bcc2f81eb91be38e86679e" Oct 30 00:59:22.804317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.804295 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:59:22.849206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.849112 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:59:22.874201 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874175 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-kube-api-access-dw4f7\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-kube-api-access-dw4f7\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.874397 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874225 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-envoy\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.874586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874561 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-istio-token\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-istio-token\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.874676 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874625 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-token\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.874735 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874696 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-data\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.874862 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874752 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-kube-api-access-s27ph\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-kube-api-access-s27ph\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.874862 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874806 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-podinfo\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.874862 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874847 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7f16c3-7f67-4664-a08d-2a088130b04b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7f16c3-7f67-4664-a08d-2a088130b04b-istiod-ca-cert\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.875043 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874918 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-credential-socket\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.875043 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.874968 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6392db76-3331-49b5-9bf3-055e24d0f203-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6392db76-3331-49b5-9bf3-055e24d0f203-istiod-ca-cert\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.875152 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875086 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-certs\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.875204 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875154 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-envoy\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.875257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875217 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6392db76-3331-49b5-9bf3-055e24d0f203-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6392db76-3331-49b5-9bf3-055e24d0f203-istio-podinfo\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.875257 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875244 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-data\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.875598 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875536 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-data" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.875702 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875618 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-certs" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.875754 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875741 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-credential-socket" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.875860 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875271 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-socket\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.875945 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875886 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-socket\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.875945 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875922 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-certs\") pod \"6392db76-3331-49b5-9bf3-055e24d0f203\" (UID: \"6392db76-3331-49b5-9bf3-055e24d0f203\") " Oct 30 00:59:22.876059 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.875997 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7f16c3-7f67-4664-a08d-2a088130b04b-istiod-ca-cert" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:22.876059 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.876045 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-credential-socket\") pod \"8e7f16c3-7f67-4664-a08d-2a088130b04b\" (UID: \"8e7f16c3-7f67-4664-a08d-2a088130b04b\") " Oct 30 00:59:22.876163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.876104 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6392db76-3331-49b5-9bf3-055e24d0f203-istiod-ca-cert" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:22.876309 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.876280 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-data" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.876412 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.876392 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-socket" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.876638 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.876613 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-certs" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.876818 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.876798 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-socket" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.876974 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.876958 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-credential-socket" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.877153 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877023 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877153 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877153 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877166 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877181 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877194 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877207 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877221 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7f16c3-7f67-4664-a08d-2a088130b04b-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877232 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.877295 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.877245 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6392db76-3331-49b5-9bf3-055e24d0f203-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.880906 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.880863 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-envoy" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.881300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.881261 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-kube-api-access-s27ph" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "kube-api-access-s27ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:22.881533 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.881494 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-podinfo" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:59:22.882828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.882796 4905 patch_prober.go:29] interesting pod/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:59:22.882924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.882851 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:59:22.883306 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.883286 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-istio-token" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:22.884195 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.884173 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6392db76-3331-49b5-9bf3-055e24d0f203-istio-podinfo" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:59:22.884928 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.884902 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-envoy" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:22.885244 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.885219 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-kube-api-access-dw4f7" pod "6392db76-3331-49b5-9bf3-055e24d0f203" (UID: "6392db76-3331-49b5-9bf3-055e24d0f203"). InnerVolumeSpecName "kube-api-access-dw4f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:22.886306 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.886282 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-token" pod "8e7f16c3-7f67-4664-a08d-2a088130b04b" (UID: "8e7f16c3-7f67-4664-a08d-2a088130b04b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980217 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980261 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6392db76-3331-49b5-9bf3-055e24d0f203-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980276 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7f16c3-7f67-4664-a08d-2a088130b04b-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980292 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw4f7\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-kube-api-access-dw4f7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980306 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6392db76-3331-49b5-9bf3-055e24d0f203-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980319 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6392db76-3331-49b5-9bf3-055e24d0f203-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980332 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980344 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s27ph\" (UniqueName: \"kubernetes.io/projected/8e7f16c3-7f67-4664-a08d-2a088130b04b-kube-api-access-s27ph\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:22.982542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:22.980358 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7f16c3-7f67-4664-a08d-2a088130b04b-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:23.574287 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.574259 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" Oct 30 00:59:23.574757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.574297 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62" event={"ID":"8e7f16c3-7f67-4664-a08d-2a088130b04b","Type":"ContainerDied","Data":"b913b8a0e2a70f9734c304c70ef0e591df13f7d4147b76f4a7cc63844b3d74b2"} Oct 30 00:59:23.574757 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.574368 4905 scope.go:122] "RemoveContainer" containerID="750979940b14d6a59ab9a8189a2c2d8f7444cccced4b73ae9460cbff0031284a" Oct 30 00:59:23.576233 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.576212 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" Oct 30 00:59:23.576357 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.576233 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5" event={"ID":"6392db76-3331-49b5-9bf3-055e24d0f203","Type":"ContainerDied","Data":"1d29649c4bb1087132d0325b77613026ae79c3c3047dba19f93e19b2e368e345"} Oct 30 00:59:23.585968 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.585952 4905 scope.go:122] "RemoveContainer" containerID="c27235258f8a83d4327f15f57afe2133fd5850c7e584c5cdabef56781cc24d9a" Oct 30 00:59:23.634431 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.634405 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5"] Oct 30 00:59:23.647609 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.647563 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-lcbc5"] Oct 30 00:59:23.673254 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.673231 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62"] Oct 30 00:59:23.677904 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.677879 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-baf5781e-openshift-default-7f84bb5cff-jhz62"] Oct 30 00:59:23.887755 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.887683 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:59:23.888876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:23.888851 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:59:25.191782 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:25.191753 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" path="/var/lib/kubelet/pods/6392db76-3331-49b5-9bf3-055e24d0f203/volumes" Oct 30 00:59:25.192266 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:25.192252 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" path="/var/lib/kubelet/pods/8e7f16c3-7f67-4664-a08d-2a088130b04b/volumes" Oct 30 00:59:25.241047 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:25.241008 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs"] Oct 30 00:59:25.263411 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:25.263376 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm"] Oct 30 00:59:26.380581 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.379162 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk"] Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380675 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380696 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380728 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380736 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380767 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380774 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380795 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380802 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380853 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" containerName="revoke-keys" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.380861 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" containerName="revoke-keys" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.381007 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.381022 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7f16c3-7f67-4664-a08d-2a088130b04b" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.381036 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" containerName="revoke-keys" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.381046 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" Oct 30 00:59:26.381065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.381059 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6392db76-3331-49b5-9bf3-055e24d0f203" containerName="istio-proxy" Oct 30 00:59:26.387458 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.387425 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.401663 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.397677 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk"] Oct 30 00:59:26.414988 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.414697 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ca324d-openshift-default-dockercfg-w2cst\"" Oct 30 00:59:26.448137 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.448107 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istiod-ca-cert\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.448375 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.448356 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-podinfo\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.449430 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.449401 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfvt\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-kube-api-access-vxfvt\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.449657 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.449641 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.449785 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.449771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-certs\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.449926 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.449909 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-data\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.450014 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.450002 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-token\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.450117 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.450105 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-envoy\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.450345 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.450329 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-credential-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.562929 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.562886 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.565927 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.565901 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-certs\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.566300 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.566279 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-data\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.566435 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.566420 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-token\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.566625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.566606 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-envoy\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.566921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.566906 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-credential-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.567260 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.567239 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istiod-ca-cert\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.567426 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.567411 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-podinfo\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.567670 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.567653 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfvt\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-kube-api-access-vxfvt\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.568349 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.565823 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.568902 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.568806 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-certs\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.571599 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.570334 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istiod-ca-cert\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.571755 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.570671 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-credential-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.571851 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.571534 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-data\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.608827 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.594293 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-token\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.609443 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.609413 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-envoy\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.639625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.616220 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-podinfo\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.639625 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.618896 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfvt\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-kube-api-access-vxfvt\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:26.719542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:26.718138 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:27.270298 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:59:27.270271 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ccc4f5_f71a_4ca8_9bb2_eb34a4eb2290.slice/crio-407dcea5d7bf089aeea44d1b47f534b336688a3155a97de043e9a98a2ccb146e WatchSource:0}: Error finding container 407dcea5d7bf089aeea44d1b47f534b336688a3155a97de043e9a98a2ccb146e: Status 404 returned error can't find the container with id 407dcea5d7bf089aeea44d1b47f534b336688a3155a97de043e9a98a2ccb146e Oct 30 00:59:27.272698 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.272669 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk"] Oct 30 00:59:27.272829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.272693 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:59:27.272829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.272763 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:59:27.272829 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.272807 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 00:59:27.697613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.697498 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" event={"ID":"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290","Type":"ContainerStarted","Data":"9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7"} Oct 30 00:59:27.697613 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.697584 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" event={"ID":"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290","Type":"ContainerStarted","Data":"407dcea5d7bf089aeea44d1b47f534b336688a3155a97de043e9a98a2ccb146e"} Oct 30 00:59:27.719259 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.719233 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:27.744648 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:27.744606 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podStartSLOduration=1.744594531 podStartE2EDuration="1.744594531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:27.739304997 +0000 UTC m=+1252.682478189" watchObservedRunningTime="2026-10-30 00:59:27.744594531 +0000 UTC m=+1252.687767722" Oct 30 00:59:28.776138 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:28.776109 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:29.714828 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:29.714791 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:29.715641 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:29.715619 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 00:59:30.241316 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.241275 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" podUID="a9b5e1c2-af98-46df-9243-c0492304cb15" containerName="payload-processing" containerID="cri-o://0f5ffa99f5590a0e73f0a5a8e9ff5f0f787029f27cc6c2fb3462c41e9c53295d" gracePeriod=25 Oct 30 00:59:30.264660 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.264584 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" podUID="5a7f26fb-51ef-4367-88ea-44e43b1fc214" containerName="payload-pre-processing" containerID="cri-o://21daf6fbdb87d9a8d7cfbe9b4e8a213837de6654d6809d20acf06f38cfd91f08" gracePeriod=25 Oct 30 00:59:30.728706 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.728679 4905 generic.go:356] "Generic (PLEG): container finished" podID="5a7f26fb-51ef-4367-88ea-44e43b1fc214" containerID="21daf6fbdb87d9a8d7cfbe9b4e8a213837de6654d6809d20acf06f38cfd91f08" exitCode=0 Oct 30 00:59:30.728854 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.728796 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" event={"ID":"5a7f26fb-51ef-4367-88ea-44e43b1fc214","Type":"ContainerDied","Data":"21daf6fbdb87d9a8d7cfbe9b4e8a213837de6654d6809d20acf06f38cfd91f08"} Oct 30 00:59:30.731143 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.731063 4905 generic.go:356] "Generic (PLEG): container finished" podID="a9b5e1c2-af98-46df-9243-c0492304cb15" containerID="0f5ffa99f5590a0e73f0a5a8e9ff5f0f787029f27cc6c2fb3462c41e9c53295d" exitCode=0 Oct 30 00:59:30.732802 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.732663 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" event={"ID":"a9b5e1c2-af98-46df-9243-c0492304cb15","Type":"ContainerDied","Data":"0f5ffa99f5590a0e73f0a5a8e9ff5f0f787029f27cc6c2fb3462c41e9c53295d"} Oct 30 00:59:30.732802 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.732717 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" event={"ID":"a9b5e1c2-af98-46df-9243-c0492304cb15","Type":"ContainerDied","Data":"6801339bd5bbde3e8ac00b9329690b1f7a548e4ab1d5edf1ee0c03c95e37381a"} Oct 30 00:59:30.732802 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.732742 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6801339bd5bbde3e8ac00b9329690b1f7a548e4ab1d5edf1ee0c03c95e37381a" Oct 30 00:59:30.741069 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.741055 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:59:30.780055 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.780035 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:59:30.849183 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.849104 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7f26fb-51ef-4367-88ea-44e43b1fc214-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a7f26fb-51ef-4367-88ea-44e43b1fc214-plugins-config-volume\") pod \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " Oct 30 00:59:30.849183 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.849167 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b5e1c2-af98-46df-9243-c0492304cb15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b5e1c2-af98-46df-9243-c0492304cb15-plugins-config-volume\") pod \"a9b5e1c2-af98-46df-9243-c0492304cb15\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " Oct 30 00:59:30.849389 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.849361 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7f26fb-51ef-4367-88ea-44e43b1fc214-kube-api-access-7z5gq\" (UniqueName: \"kubernetes.io/projected/5a7f26fb-51ef-4367-88ea-44e43b1fc214-kube-api-access-7z5gq\") pod \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\" (UID: \"5a7f26fb-51ef-4367-88ea-44e43b1fc214\") " Oct 30 00:59:30.849456 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.849439 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b5e1c2-af98-46df-9243-c0492304cb15-kube-api-access-qmh9c\" (UniqueName: \"kubernetes.io/projected/a9b5e1c2-af98-46df-9243-c0492304cb15-kube-api-access-qmh9c\") pod \"a9b5e1c2-af98-46df-9243-c0492304cb15\" (UID: \"a9b5e1c2-af98-46df-9243-c0492304cb15\") " Oct 30 00:59:30.850817 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.850787 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7f26fb-51ef-4367-88ea-44e43b1fc214-plugins-config-volume" pod "5a7f26fb-51ef-4367-88ea-44e43b1fc214" (UID: "5a7f26fb-51ef-4367-88ea-44e43b1fc214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:30.851987 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.851960 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b5e1c2-af98-46df-9243-c0492304cb15-plugins-config-volume" pod "a9b5e1c2-af98-46df-9243-c0492304cb15" (UID: "a9b5e1c2-af98-46df-9243-c0492304cb15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:30.853586 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.853472 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b5e1c2-af98-46df-9243-c0492304cb15-kube-api-access-qmh9c" pod "a9b5e1c2-af98-46df-9243-c0492304cb15" (UID: "a9b5e1c2-af98-46df-9243-c0492304cb15"). InnerVolumeSpecName "kube-api-access-qmh9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:30.854943 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.854918 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7f26fb-51ef-4367-88ea-44e43b1fc214-kube-api-access-7z5gq" pod "5a7f26fb-51ef-4367-88ea-44e43b1fc214" (UID: "5a7f26fb-51ef-4367-88ea-44e43b1fc214"). InnerVolumeSpecName "kube-api-access-7z5gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:30.951726 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.951694 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a7f26fb-51ef-4367-88ea-44e43b1fc214-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:30.951726 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.951720 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b5e1c2-af98-46df-9243-c0492304cb15-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:30.951726 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.951729 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z5gq\" (UniqueName: \"kubernetes.io/projected/5a7f26fb-51ef-4367-88ea-44e43b1fc214-kube-api-access-7z5gq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:30.951924 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:30.951737 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmh9c\" (UniqueName: \"kubernetes.io/projected/a9b5e1c2-af98-46df-9243-c0492304cb15-kube-api-access-qmh9c\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:31.742806 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.742779 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" Oct 30 00:59:31.743206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.742820 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm" event={"ID":"5a7f26fb-51ef-4367-88ea-44e43b1fc214","Type":"ContainerDied","Data":"3e10379437a8996963368b1abf5fda08f489875a48367ce73f48f84eaf63e741"} Oct 30 00:59:31.743206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.742881 4905 scope.go:122] "RemoveContainer" containerID="21daf6fbdb87d9a8d7cfbe9b4e8a213837de6654d6809d20acf06f38cfd91f08" Oct 30 00:59:31.743206 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.742941 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs" Oct 30 00:59:31.819608 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.819579 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm"] Oct 30 00:59:31.833556 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.833513 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f146c27-7498757655-8nftm"] Oct 30 00:59:31.859651 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.858752 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs"] Oct 30 00:59:31.864232 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:31.864210 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f146c27-6885648654-s6lcs"] Oct 30 00:59:32.193880 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.193807 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf"] Oct 30 00:59:32.194488 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.194474 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a9b5e1c2-af98-46df-9243-c0492304cb15" containerName="payload-processing" Oct 30 00:59:32.194576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.194490 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b5e1c2-af98-46df-9243-c0492304cb15" containerName="payload-processing" Oct 30 00:59:32.194576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.194505 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5a7f26fb-51ef-4367-88ea-44e43b1fc214" containerName="payload-pre-processing" Oct 30 00:59:32.194576 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.194510 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7f26fb-51ef-4367-88ea-44e43b1fc214" containerName="payload-pre-processing" Oct 30 00:59:32.194689 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.194664 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b5e1c2-af98-46df-9243-c0492304cb15" containerName="payload-processing" Oct 30 00:59:32.194689 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.194676 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7f26fb-51ef-4367-88ea-44e43b1fc214" containerName="payload-pre-processing" Oct 30 00:59:32.199842 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.199824 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.215595 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.215567 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ca324d\"" Oct 30 00:59:32.215804 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.215789 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ca324d-dockercfg-xfchn\"" Oct 30 00:59:32.216688 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.216672 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf"] Oct 30 00:59:32.221966 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.221940 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd71e90d-0219-4f08-8b91-15302757b1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.222126 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.222106 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsz2h\" (UniqueName: \"kubernetes.io/projected/bd71e90d-0219-4f08-8b91-15302757b1c1-kube-api-access-hsz2h\") pod \"payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.353547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.353169 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsz2h\" (UniqueName: \"kubernetes.io/projected/bd71e90d-0219-4f08-8b91-15302757b1c1-kube-api-access-hsz2h\") pod \"payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.353547 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.353541 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd71e90d-0219-4f08-8b91-15302757b1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.358602 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.354773 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd71e90d-0219-4f08-8b91-15302757b1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.578541 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.578490 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsz2h\" (UniqueName: \"kubernetes.io/projected/bd71e90d-0219-4f08-8b91-15302757b1c1-kube-api-access-hsz2h\") pod \"payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.594410 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.594360 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm"] Oct 30 00:59:32.655114 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.655078 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm"] Oct 30 00:59:32.655312 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.655273 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.664253 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.664220 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:32.827036 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.826996 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlx95\" (UniqueName: \"kubernetes.io/projected/6226c647-d9fb-45a4-8364-4ab4550601e6-kube-api-access-zlx95\") pod \"payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.827432 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.827062 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6226c647-d9fb-45a4-8364-4ab4550601e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.929301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.929210 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlx95\" (UniqueName: \"kubernetes.io/projected/6226c647-d9fb-45a4-8364-4ab4550601e6-kube-api-access-zlx95\") pod \"payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.929301 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.929267 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6226c647-d9fb-45a4-8364-4ab4550601e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.930290 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.930267 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6226c647-d9fb-45a4-8364-4ab4550601e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.950396 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.950373 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlx95\" (UniqueName: \"kubernetes.io/projected/6226c647-d9fb-45a4-8364-4ab4550601e6-kube-api-access-zlx95\") pod \"payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.980085 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.980060 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:32.980317 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:59:32.980294 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd71e90d_0219_4f08_8b91_15302757b1c1.slice/crio-9c9d65fd8f4606d5d67662697cba83750fd5e18fa08e37d033c1a89daefe8595 WatchSource:0}: Error finding container 9c9d65fd8f4606d5d67662697cba83750fd5e18fa08e37d033c1a89daefe8595: Status 404 returned error can't find the container with id 9c9d65fd8f4606d5d67662697cba83750fd5e18fa08e37d033c1a89daefe8595 Oct 30 00:59:32.985799 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:32.985319 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf"] Oct 30 00:59:33.192395 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.192310 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7f26fb-51ef-4367-88ea-44e43b1fc214" path="/var/lib/kubelet/pods/5a7f26fb-51ef-4367-88ea-44e43b1fc214/volumes" Oct 30 00:59:33.192935 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.192911 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b5e1c2-af98-46df-9243-c0492304cb15" path="/var/lib/kubelet/pods/a9b5e1c2-af98-46df-9243-c0492304cb15/volumes" Oct 30 00:59:33.291548 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:59:33.291507 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6226c647_d9fb_45a4_8364_4ab4550601e6.slice/crio-cbb91432e1489123a96942556bed1351dd2c2017c4e4645507c240f980ba001a WatchSource:0}: Error finding container cbb91432e1489123a96942556bed1351dd2c2017c4e4645507c240f980ba001a: Status 404 returned error can't find the container with id cbb91432e1489123a96942556bed1351dd2c2017c4e4645507c240f980ba001a Oct 30 00:59:33.299281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.299253 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm"] Oct 30 00:59:33.776467 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.776426 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" event={"ID":"bd71e90d-0219-4f08-8b91-15302757b1c1","Type":"ContainerStarted","Data":"51c1007b0a9ab2f40874c236a14cacdf691ab8be86df76bb5fb93cb71c84fffc"} Oct 30 00:59:33.776656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.776485 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" event={"ID":"bd71e90d-0219-4f08-8b91-15302757b1c1","Type":"ContainerStarted","Data":"9c9d65fd8f4606d5d67662697cba83750fd5e18fa08e37d033c1a89daefe8595"} Oct 30 00:59:33.776656 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.776508 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:33.778283 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.778256 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" event={"ID":"6226c647-d9fb-45a4-8364-4ab4550601e6","Type":"ContainerStarted","Data":"07da7c68a55207de877e140534ae0516148fb9fcb4ddeeede66ec66bd69b5eda"} Oct 30 00:59:33.778422 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.778308 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" event={"ID":"6226c647-d9fb-45a4-8364-4ab4550601e6","Type":"ContainerStarted","Data":"cbb91432e1489123a96942556bed1351dd2c2017c4e4645507c240f980ba001a"} Oct 30 00:59:33.778422 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.778337 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:33.906105 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.906043 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" podStartSLOduration=1.906021253 podStartE2EDuration="1.906021253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:33.89595566 +0000 UTC m=+1258.839128856" watchObservedRunningTime="2026-10-30 00:59:33.906021253 +0000 UTC m=+1258.849194446" Oct 30 00:59:33.937030 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:33.936665 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" podStartSLOduration=1.936645387 podStartE2EDuration="1.936645387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:33.936024249 +0000 UTC m=+1258.879197441" watchObservedRunningTime="2026-10-30 00:59:33.936645387 +0000 UTC m=+1258.879818579" Oct 30 00:59:36.896408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:36.896378 4905 scope.go:122] "RemoveContainer" containerID="322f060e9e263bc9e4642e22e3282d895cfdefbdb96b8ce2b3d93719003574c1" Oct 30 00:59:36.908062 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:36.908040 4905 scope.go:122] "RemoveContainer" containerID="cd3e9a9fc3fb4a3180d92333d3dd2f4487c7211823d6db5157848c0cc4dba0ef" Oct 30 00:59:36.918205 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:36.918187 4905 scope.go:122] "RemoveContainer" containerID="efb48a74cf2ccbf3abd5b65c3f6814c6a81f5c424612a76269933bc7ab0ed380" Oct 30 00:59:42.018633 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:42.018596 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr"] Oct 30 00:59:42.031768 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:42.031722 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f146c27-rv2cr"] Oct 30 00:59:42.418703 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:59:42.418621 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 00:59:43.195567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:43.195512 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ccd9eb-7c63-4a34-8bfc-d6db54926cda" path="/var/lib/kubelet/pods/e5ccd9eb-7c63-4a34-8bfc-d6db54926cda/volumes" Oct 30 00:59:43.535353 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:43.535316 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9"] Oct 30 00:59:43.535638 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:43.535585 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" containerID="cri-o://985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186" gracePeriod=30 Oct 30 00:59:44.789743 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:44.789714 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 00:59:44.790213 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:44.790126 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 00:59:47.774219 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.774187 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2"] Oct 30 00:59:47.780883 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.780864 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:47.800196 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.800174 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2"] Oct 30 00:59:47.806845 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.806821 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:59:47.807003 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.806865 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:59:47.807003 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.806905 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:59:47.810372 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.810344 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:59:47.952567 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.952508 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:47.952759 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.952691 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8fc\" (UniqueName: \"kubernetes.io/projected/1a79bc14-c169-4816-8ec8-ea7df7bc2125-kube-api-access-fr8fc\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:47.952824 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:47.952789 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.070597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.054838 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.070597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.055101 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.070597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.055255 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8fc\" (UniqueName: \"kubernetes.io/projected/1a79bc14-c169-4816-8ec8-ea7df7bc2125-kube-api-access-fr8fc\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.070597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.055610 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.095539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.094971 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.118539 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.102798 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8fc\" (UniqueName: \"kubernetes.io/projected/1a79bc14-c169-4816-8ec8-ea7df7bc2125-kube-api-access-fr8fc\") pod \"maas-api-revoke-sub-f3e98016-6c7l2\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.398317 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.398234 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:48.704095 ip-10-0-130-171 kubenswrapper[4905]: W1030 00:59:48.704067 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a79bc14_c169_4816_8ec8_ea7df7bc2125.slice/crio-9bcf67aa87b517d3c3f2c5e51d97ed90f231ead096e9089dda9ffc20a8c00fb2 WatchSource:0}: Error finding container 9bcf67aa87b517d3c3f2c5e51d97ed90f231ead096e9089dda9ffc20a8c00fb2: Status 404 returned error can't find the container with id 9bcf67aa87b517d3c3f2c5e51d97ed90f231ead096e9089dda9ffc20a8c00fb2 Oct 30 00:59:48.707826 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.707804 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2"] Oct 30 00:59:48.819198 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.819174 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:59:48.970302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.969829 4905 generic.go:356] "Generic (PLEG): container finished" podID="9eb00637-ca24-4047-af6b-9535ead35880" containerID="985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186" exitCode=0 Oct 30 00:59:48.970302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.969897 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" Oct 30 00:59:48.970302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.970044 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" event={"ID":"9eb00637-ca24-4047-af6b-9535ead35880","Type":"ContainerDied","Data":"985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186"} Oct 30 00:59:48.970302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.970092 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9" event={"ID":"9eb00637-ca24-4047-af6b-9535ead35880","Type":"ContainerDied","Data":"60103cb322bffdba68d1ecd7760d36a6ba4cb094102f461b3a76c57532b4a953"} Oct 30 00:59:48.970302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.970122 4905 scope.go:122] "RemoveContainer" containerID="985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186" Oct 30 00:59:48.972425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.972383 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" event={"ID":"1a79bc14-c169-4816-8ec8-ea7df7bc2125","Type":"ContainerStarted","Data":"9bcf67aa87b517d3c3f2c5e51d97ed90f231ead096e9089dda9ffc20a8c00fb2"} Oct 30 00:59:48.975302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975201 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-socket\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975251 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-istio-token\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-istio-token\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975302 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975278 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb00637-ca24-4047-af6b-9535ead35880-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb00637-ca24-4047-af6b-9535ead35880-istiod-ca-cert\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975392 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-envoy\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975553 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-credential-socket\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975582 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975585 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-certs\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975780 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975609 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eb00637-ca24-4047-af6b-9535ead35880-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb00637-ca24-4047-af6b-9535ead35880-istio-podinfo\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975780 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975629 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-data\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.975780 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.975772 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-kube-api-access-jv5hj\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-kube-api-access-jv5hj\") pod \"9eb00637-ca24-4047-af6b-9535ead35880\" (UID: \"9eb00637-ca24-4047-af6b-9535ead35880\") " Oct 30 00:59:48.976106 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.976070 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-socket" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:48.976279 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.976254 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-credential-socket" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:48.976395 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.976375 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:48.976450 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.976388 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:48.976493 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.976446 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-certs" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:48.976918 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.976880 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-data" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:48.977243 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.977218 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb00637-ca24-4047-af6b-9535ead35880-istiod-ca-cert" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:48.978841 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.978818 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-istio-token" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:48.979065 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.979045 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-kube-api-access-jv5hj" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "kube-api-access-jv5hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:48.979161 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.979144 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-envoy" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:48.979665 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.979633 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb00637-ca24-4047-af6b-9535ead35880-istio-podinfo" pod "9eb00637-ca24-4047-af6b-9535ead35880" (UID: "9eb00637-ca24-4047-af6b-9535ead35880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:59:48.983867 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.983850 4905 scope.go:122] "RemoveContainer" containerID="44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927" Oct 30 00:59:48.994494 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.994218 4905 scope.go:122] "RemoveContainer" containerID="985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186" Oct 30 00:59:48.994825 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:59:48.994804 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186\": container with ID starting with 985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186 not found: ID does not exist" containerID="985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186" Oct 30 00:59:48.994921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.994832 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186"} err="failed to get container status \"985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186\": rpc error: code = NotFound desc = could not find container \"985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186\": container with ID starting with 985b5af29db556387b7235b9ca7a843316886c1286efdadea7a29f13a41e4186 not found: ID does not exist" Oct 30 00:59:48.994921 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.994850 4905 scope.go:122] "RemoveContainer" containerID="44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927" Oct 30 00:59:48.995161 ip-10-0-130-171 kubenswrapper[4905]: E1030 00:59:48.995133 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927\": container with ID starting with 44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927 not found: ID does not exist" containerID="44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927" Oct 30 00:59:48.995214 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:48.995168 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927"} err="failed to get container status \"44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927\": rpc error: code = NotFound desc = could not find container \"44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927\": container with ID starting with 44757e6f18933e26ef025a8fc6f330297d1d8cfbd8186446ecc474b281708927 not found: ID does not exist" Oct 30 00:59:49.077687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.077654 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:49.077687 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.077681 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb00637-ca24-4047-af6b-9535ead35880-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:49.077847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.077694 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:49.077847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.077706 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:49.077847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.077717 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb00637-ca24-4047-af6b-9535ead35880-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:49.077847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.077729 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb00637-ca24-4047-af6b-9535ead35880-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:49.077847 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.077741 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv5hj\" (UniqueName: \"kubernetes.io/projected/9eb00637-ca24-4047-af6b-9535ead35880-kube-api-access-jv5hj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:49.316600 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.316508 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9"] Oct 30 00:59:49.321658 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.321632 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5f146c27-openshift-default-774b6845bf-4rph9"] Oct 30 00:59:49.985998 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.985971 4905 generic.go:356] "Generic (PLEG): container finished" podID="1a79bc14-c169-4816-8ec8-ea7df7bc2125" containerID="b48967f80fb00b72a6008b732bd854c63c2680a27ba583207420d194a75c777c" exitCode=0 Oct 30 00:59:49.986371 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:49.986057 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" event={"ID":"1a79bc14-c169-4816-8ec8-ea7df7bc2125","Type":"ContainerDied","Data":"b48967f80fb00b72a6008b732bd854c63c2680a27ba583207420d194a75c777c"} Oct 30 00:59:51.469553 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.455879 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb00637-ca24-4047-af6b-9535ead35880" path="/var/lib/kubelet/pods/9eb00637-ca24-4047-af6b-9535ead35880/volumes" Oct 30 00:59:51.499307 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.499282 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:51.620728 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.620702 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-openshift-ca\") pod \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " Oct 30 00:59:51.620917 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.620903 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-xks-ca\") pod \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " Oct 30 00:59:51.621018 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.620993 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a79bc14-c169-4816-8ec8-ea7df7bc2125-kube-api-access-fr8fc\" (UniqueName: \"kubernetes.io/projected/1a79bc14-c169-4816-8ec8-ea7df7bc2125-kube-api-access-fr8fc\") pod \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\" (UID: \"1a79bc14-c169-4816-8ec8-ea7df7bc2125\") " Oct 30 00:59:51.621109 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.621088 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-openshift-ca" pod "1a79bc14-c169-4816-8ec8-ea7df7bc2125" (UID: "1a79bc14-c169-4816-8ec8-ea7df7bc2125"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:51.621406 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.621391 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:51.623164 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.623142 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-xks-ca" pod "1a79bc14-c169-4816-8ec8-ea7df7bc2125" (UID: "1a79bc14-c169-4816-8ec8-ea7df7bc2125"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:51.623250 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.623158 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a79bc14-c169-4816-8ec8-ea7df7bc2125-kube-api-access-fr8fc" pod "1a79bc14-c169-4816-8ec8-ea7df7bc2125" (UID: "1a79bc14-c169-4816-8ec8-ea7df7bc2125"). InnerVolumeSpecName "kube-api-access-fr8fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:51.722952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.722891 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8fc\" (UniqueName: \"kubernetes.io/projected/1a79bc14-c169-4816-8ec8-ea7df7bc2125-kube-api-access-fr8fc\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:51.722952 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:51.722918 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a79bc14-c169-4816-8ec8-ea7df7bc2125-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.006292 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:52.006256 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" event={"ID":"1a79bc14-c169-4816-8ec8-ea7df7bc2125","Type":"ContainerDied","Data":"9bcf67aa87b517d3c3f2c5e51d97ed90f231ead096e9089dda9ffc20a8c00fb2"} Oct 30 00:59:52.006468 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:52.006314 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bcf67aa87b517d3c3f2c5e51d97ed90f231ead096e9089dda9ffc20a8c00fb2" Oct 30 00:59:52.006468 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:52.006270 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2" Oct 30 00:59:56.256546 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.255952 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw"] Oct 30 00:59:56.324072 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.324040 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" Oct 30 00:59:56.339858 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.339830 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" Oct 30 00:59:56.340068 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.340055 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" Oct 30 00:59:56.340163 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.340152 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" Oct 30 00:59:56.340281 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.340271 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1a79bc14-c169-4816-8ec8-ea7df7bc2125" containerName="revoke-keys" Oct 30 00:59:56.340358 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.340347 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a79bc14-c169-4816-8ec8-ea7df7bc2125" containerName="revoke-keys" Oct 30 00:59:56.359938 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.359905 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" Oct 30 00:59:56.360159 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.360146 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb00637-ca24-4047-af6b-9535ead35880" containerName="istio-proxy" Oct 30 00:59:56.360255 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.360243 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a79bc14-c169-4816-8ec8-ea7df7bc2125" containerName="revoke-keys" Oct 30 00:59:56.621601 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.569863 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw"] Oct 30 00:59:56.736550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.736345 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.736550 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.736439 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.763542 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.736820 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhb8\" (UniqueName: \"kubernetes.io/projected/5d0aa9d6-0277-4935-87c5-40538906eecf-kube-api-access-hdhb8\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.805544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.804752 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.831359 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.831334 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:59:56.845637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.844902 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:59:56.845637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.844970 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 00:59:56.845637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.845058 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:59:56.850558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.846308 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhb8\" (UniqueName: \"kubernetes.io/projected/5d0aa9d6-0277-4935-87c5-40538906eecf-kube-api-access-hdhb8\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.850558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.846403 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.850558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.846463 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.850558 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.847444 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.905425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.880465 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.905425 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.892959 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhb8\" (UniqueName: \"kubernetes.io/projected/5d0aa9d6-0277-4935-87c5-40538906eecf-kube-api-access-hdhb8\") pod \"maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:56.975544 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:56.974762 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 00:59:57.325543 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:57.310564 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:57.974597 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:57.974196 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 00:59:58.077919 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.053662 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:58.150551 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.129925 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:58.291642 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.263802 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 00:59:58.320637 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.320601 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:58.321499 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.321477 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:59:58.415916 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.414998 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:58.434876 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.421983 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:58.720408 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.720302 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw"] Oct 30 00:59:58.721563 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.721485 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-g5ltm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:59:58.721697 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.721577 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:58.801689 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:58.799723 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 00:59:59.809122 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:59.809086 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" event={"ID":"5d0aa9d6-0277-4935-87c5-40538906eecf","Type":"ContainerStarted","Data":"9bdd584e1da2586ad691312f37d211e2f6fdbb2e2c7270fd68a0445464ffe5fe"} Oct 30 00:59:59.809622 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:59.809144 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" event={"ID":"5d0aa9d6-0277-4935-87c5-40538906eecf","Type":"ContainerStarted","Data":"f45c7c28a2913924ac6595a37a9e86929fc17f3e46316d29b3a198ab7eca5ea8"} Oct 30 00:59:59.856477 ip-10-0-130-171 kubenswrapper[4905]: I1030 00:59:59.856419 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" podStartSLOduration=3.856401481 podStartE2EDuration="3.856401481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:59.849104024 +0000 UTC m=+1284.792277217" watchObservedRunningTime="2026-10-30 00:59:59.856401481 +0000 UTC m=+1284.799574675" Oct 30 01:00:00.149968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.149885 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq"] Oct 30 01:00:00.155869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.155843 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r"] Oct 30 01:00:00.156126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.155963 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.161366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.161343 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v"] Oct 30 01:00:00.161497 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.161474 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.166229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.166208 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.167923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.167900 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45"] Oct 30 01:00:00.176803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.176634 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.182583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.180431 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv"] Oct 30 01:00:00.186061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.186041 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr"] Oct 30 01:00:00.196030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.192791 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.196872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.196767 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq"] Oct 30 01:00:00.196872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.196868 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.201468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.200761 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r"] Oct 30 01:00:00.209165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.206689 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v"] Oct 30 01:00:00.212502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.212482 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45"] Oct 30 01:00:00.219867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.219830 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv"] Oct 30 01:00:00.226836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.226816 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr"] Oct 30 01:00:00.319889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.319861 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.320044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.319923 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.320128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320036 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-mdd8v\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.320128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320081 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad6109-5703-409d-8441-798c75dae1f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.320234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320215 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb2b8df-479a-417d-8105-035cb8a422a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.320330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320308 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1397a8c5-495a-4b7e-a747-e9220a1e0869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.320370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320357 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwnr\" (UniqueName: \"kubernetes.io/projected/c1ad6109-5703-409d-8441-798c75dae1f8-kube-api-access-6lwnr\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.320457 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320441 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5722f\" (UniqueName: \"kubernetes.io/projected/2eb2b8df-479a-417d-8105-035cb8a422a5-kube-api-access-5722f\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.320545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320531 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzc5\" (UniqueName: \"kubernetes.io/projected/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-kube-api-access-wdzc5\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.320605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320595 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcl5m\" (UniqueName: \"kubernetes.io/projected/1397a8c5-495a-4b7e-a747-e9220a1e0869-kube-api-access-qcl5m\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.320655 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320643 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7bg\" (UniqueName: \"kubernetes.io/projected/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-kube-api-access-4m7bg\") pod \"maas-api-key-cleanup-29888700-mdd8v\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.320693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.320663 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmg6q\" (UniqueName: \"kubernetes.io/projected/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-kube-api-access-xmg6q\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.422332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.422244 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwnr\" (UniqueName: \"kubernetes.io/projected/c1ad6109-5703-409d-8441-798c75dae1f8-kube-api-access-6lwnr\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.422332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.422319 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5722f\" (UniqueName: \"kubernetes.io/projected/2eb2b8df-479a-417d-8105-035cb8a422a5-kube-api-access-5722f\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.422629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.422430 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzc5\" (UniqueName: \"kubernetes.io/projected/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-kube-api-access-wdzc5\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.422629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.422592 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcl5m\" (UniqueName: \"kubernetes.io/projected/1397a8c5-495a-4b7e-a747-e9220a1e0869-kube-api-access-qcl5m\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.423008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.422982 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7bg\" (UniqueName: \"kubernetes.io/projected/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-kube-api-access-4m7bg\") pod \"maas-api-key-cleanup-29888700-mdd8v\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.423160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423025 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmg6q\" (UniqueName: \"kubernetes.io/projected/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-kube-api-access-xmg6q\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.423160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423123 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.423274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423181 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.423274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423233 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-mdd8v\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.423372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423273 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad6109-5703-409d-8441-798c75dae1f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.423372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423324 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb2b8df-479a-417d-8105-035cb8a422a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.423474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423382 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1397a8c5-495a-4b7e-a747-e9220a1e0869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.423933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423913 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.424017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423979 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1397a8c5-495a-4b7e-a747-e9220a1e0869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.424017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.423980 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.424397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.424380 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad6109-5703-409d-8441-798c75dae1f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.424455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.424439 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb2b8df-479a-417d-8105-035cb8a422a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.424656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.424631 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-mdd8v\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.444025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.443693 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzc5\" (UniqueName: \"kubernetes.io/projected/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-kube-api-access-wdzc5\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.444025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.443838 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwnr\" (UniqueName: \"kubernetes.io/projected/c1ad6109-5703-409d-8441-798c75dae1f8-kube-api-access-6lwnr\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.444025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.443937 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5722f\" (UniqueName: \"kubernetes.io/projected/2eb2b8df-479a-417d-8105-035cb8a422a5-kube-api-access-5722f\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.444320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.444298 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7bg\" (UniqueName: \"kubernetes.io/projected/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-kube-api-access-4m7bg\") pod \"maas-api-key-cleanup-29888700-mdd8v\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.445666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.445642 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcl5m\" (UniqueName: \"kubernetes.io/projected/1397a8c5-495a-4b7e-a747-e9220a1e0869-kube-api-access-qcl5m\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.455933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.455901 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmg6q\" (UniqueName: \"kubernetes.io/projected/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-kube-api-access-xmg6q\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.481460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.481435 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:00.510492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.510458 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:00.523990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.523967 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:00.577841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.577077 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:00.577841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.577365 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:00.577841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.577583 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:00.808070 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:00.808026 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb2b8df_479a_417d_8105_035cb8a422a5.slice/crio-01b758fee28b48f851ecf429af30e1c466002d4d57f4b442cad08c2cf99ceed8 WatchSource:0}: Error finding container 01b758fee28b48f851ecf429af30e1c466002d4d57f4b442cad08c2cf99ceed8: Status 404 returned error can't find the container with id 01b758fee28b48f851ecf429af30e1c466002d4d57f4b442cad08c2cf99ceed8 Oct 30 01:00:00.808850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.808813 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq"] Oct 30 01:00:00.828315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.828282 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerStarted","Data":"01b758fee28b48f851ecf429af30e1c466002d4d57f4b442cad08c2cf99ceed8"} Oct 30 01:00:00.831894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.831857 4905 generic.go:356] "Generic (PLEG): container finished" podID="5d0aa9d6-0277-4935-87c5-40538906eecf" containerID="9bdd584e1da2586ad691312f37d211e2f6fdbb2e2c7270fd68a0445464ffe5fe" exitCode=0 Oct 30 01:00:00.832029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.832011 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" event={"ID":"5d0aa9d6-0277-4935-87c5-40538906eecf","Type":"ContainerDied","Data":"9bdd584e1da2586ad691312f37d211e2f6fdbb2e2c7270fd68a0445464ffe5fe"} Oct 30 01:00:00.949707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:00.949622 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v"] Oct 30 01:00:01.018179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.018147 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv"] Oct 30 01:00:01.032104 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:01.032054 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e968a20_13f7_4bd2_8a94_ffa4d22dccb8.slice/crio-57902452152d6e97880f8c6ae0ca463d89ec64a372cb3f83f4abe2e44913b1df WatchSource:0}: Error finding container 57902452152d6e97880f8c6ae0ca463d89ec64a372cb3f83f4abe2e44913b1df: Status 404 returned error can't find the container with id 57902452152d6e97880f8c6ae0ca463d89ec64a372cb3f83f4abe2e44913b1df Oct 30 01:00:01.071658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.063241 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r"] Oct 30 01:00:01.087485 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:01.087461 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ad6109_5703_409d_8441_798c75dae1f8.slice/crio-0aad724bde5682ab580ba7ccbd20dbe3df0681aa95bf231a7c43bb406788ab93 WatchSource:0}: Error finding container 0aad724bde5682ab580ba7ccbd20dbe3df0681aa95bf231a7c43bb406788ab93: Status 404 returned error can't find the container with id 0aad724bde5682ab580ba7ccbd20dbe3df0681aa95bf231a7c43bb406788ab93 Oct 30 01:00:01.194073 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:01.194035 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455d2a35_b7c5_4b1f_8b1c_3048f14f6ec7.slice/crio-62c16d67325117168f7ff538ebf9e82a00c69d8480af2b6bf7660d7e149ba838 WatchSource:0}: Error finding container 62c16d67325117168f7ff538ebf9e82a00c69d8480af2b6bf7660d7e149ba838: Status 404 returned error can't find the container with id 62c16d67325117168f7ff538ebf9e82a00c69d8480af2b6bf7660d7e149ba838 Oct 30 01:00:01.200855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.200831 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr"] Oct 30 01:00:01.203088 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:01.203051 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1397a8c5_495a_4b7e_a747_e9220a1e0869.slice/crio-652c0054572cf9545fff3e4cfae04ceb076e2c8efc4943b40784bb5f1a4a4dac WatchSource:0}: Error finding container 652c0054572cf9545fff3e4cfae04ceb076e2c8efc4943b40784bb5f1a4a4dac: Status 404 returned error can't find the container with id 652c0054572cf9545fff3e4cfae04ceb076e2c8efc4943b40784bb5f1a4a4dac Oct 30 01:00:01.206013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.205992 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45"] Oct 30 01:00:01.845904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.845860 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerStarted","Data":"a98cabaf2034b027a931c1b2ce965f4a1f43a3de1eaf5349a69caa4dc9a0611c"} Oct 30 01:00:01.846239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.845930 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerStarted","Data":"57902452152d6e97880f8c6ae0ca463d89ec64a372cb3f83f4abe2e44913b1df"} Oct 30 01:00:01.848353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.848324 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerStarted","Data":"9f322422a0fcccf2a1d27a69eb0f988c8163802ba35f423450b7576dc9449d0d"} Oct 30 01:00:01.848471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.848375 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerStarted","Data":"62c16d67325117168f7ff538ebf9e82a00c69d8480af2b6bf7660d7e149ba838"} Oct 30 01:00:01.850551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.850313 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerStarted","Data":"5c9d17cd6806946704771a993bb5f71b732e8b696e3c5c4fa7af74788730da2d"} Oct 30 01:00:01.850551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.850371 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerStarted","Data":"d95287583de8231f07caa9fbeebbca6f27e2745633a50640a96d621372bf0b7b"} Oct 30 01:00:01.853150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.853126 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerStarted","Data":"904fc665dd9c81ef4c4b5030fb49ee166b399e3976bb615d0a16e66451d8f1f7"} Oct 30 01:00:01.855092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.855069 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerStarted","Data":"18ed400e23ed551b6f45056a9be5bdd9dad5f22489726d74a6104dd71c178b42"} Oct 30 01:00:01.855192 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.855114 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerStarted","Data":"0aad724bde5682ab580ba7ccbd20dbe3df0681aa95bf231a7c43bb406788ab93"} Oct 30 01:00:01.857052 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.857023 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" event={"ID":"1397a8c5-495a-4b7e-a747-e9220a1e0869","Type":"ContainerStarted","Data":"652c0054572cf9545fff3e4cfae04ceb076e2c8efc4943b40784bb5f1a4a4dac"} Oct 30 01:00:01.892726 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.892663 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" podStartSLOduration=1.892645941 podStartE2EDuration="1.892645941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.888565975 +0000 UTC m=+1286.831739159" watchObservedRunningTime="2026-10-30 01:00:01.892645941 +0000 UTC m=+1286.835819133" Oct 30 01:00:01.937751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.935715 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" podStartSLOduration=1.9356978360000001 podStartE2EDuration="1.935697836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.932574479 +0000 UTC m=+1286.875747683" watchObservedRunningTime="2026-10-30 01:00:01.935697836 +0000 UTC m=+1286.878871028" Oct 30 01:00:01.979163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:01.978454 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" podStartSLOduration=1.978440398 podStartE2EDuration="1.978440398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.973664459 +0000 UTC m=+1286.916837645" watchObservedRunningTime="2026-10-30 01:00:01.978440398 +0000 UTC m=+1286.921613589" Oct 30 01:00:02.024161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.024061 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" podStartSLOduration=2.024043249 podStartE2EDuration="2.024043249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.022295347 +0000 UTC m=+1286.965468538" watchObservedRunningTime="2026-10-30 01:00:02.024043249 +0000 UTC m=+1286.967216442" Oct 30 01:00:02.056118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.056092 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 01:00:02.090094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.090010 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" podStartSLOduration=2.08999301 podStartE2EDuration="2.08999301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.082282926 +0000 UTC m=+1287.025456110" watchObservedRunningTime="2026-10-30 01:00:02.08999301 +0000 UTC m=+1287.033166206" Oct 30 01:00:02.133271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.133218 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" podStartSLOduration=2.133200975 podStartE2EDuration="2.133200975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.130749575 +0000 UTC m=+1287.073922779" watchObservedRunningTime="2026-10-30 01:00:02.133200975 +0000 UTC m=+1287.076374168" Oct 30 01:00:02.157334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.157296 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-openshift-ca\") pod \"5d0aa9d6-0277-4935-87c5-40538906eecf\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " Oct 30 01:00:02.157483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.157465 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-xks-ca\") pod \"5d0aa9d6-0277-4935-87c5-40538906eecf\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " Oct 30 01:00:02.157700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.157684 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0aa9d6-0277-4935-87c5-40538906eecf-kube-api-access-hdhb8\" (UniqueName: \"kubernetes.io/projected/5d0aa9d6-0277-4935-87c5-40538906eecf-kube-api-access-hdhb8\") pod \"5d0aa9d6-0277-4935-87c5-40538906eecf\" (UID: \"5d0aa9d6-0277-4935-87c5-40538906eecf\") " Oct 30 01:00:02.158066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.158037 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-openshift-ca" pod "5d0aa9d6-0277-4935-87c5-40538906eecf" (UID: "5d0aa9d6-0277-4935-87c5-40538906eecf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:02.158387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.158365 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:02.162574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.160227 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-xks-ca" pod "5d0aa9d6-0277-4935-87c5-40538906eecf" (UID: "5d0aa9d6-0277-4935-87c5-40538906eecf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:00:02.162681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.162664 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0aa9d6-0277-4935-87c5-40538906eecf-kube-api-access-hdhb8" pod "5d0aa9d6-0277-4935-87c5-40538906eecf" (UID: "5d0aa9d6-0277-4935-87c5-40538906eecf"). InnerVolumeSpecName "kube-api-access-hdhb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:02.259895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.259859 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0aa9d6-0277-4935-87c5-40538906eecf-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:02.259895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.259894 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdhb8\" (UniqueName: \"kubernetes.io/projected/5d0aa9d6-0277-4935-87c5-40538906eecf-kube-api-access-hdhb8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:02.805877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.805843 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4"] Oct 30 01:00:02.806115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.806080 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" containerID="cri-o://d8d26e6acb18df183a97d45bd4e2120b75334e27342d6778fac363b8aca5ebbf" gracePeriod=30 Oct 30 01:00:02.820452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.820422 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm"] Oct 30 01:00:02.820712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.820685 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" containerID="cri-o://f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26" gracePeriod=30 Oct 30 01:00:02.839747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.839709 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds"] Oct 30 01:00:02.840158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.840126 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" containerID="cri-o://bc77cd748485055be227c44c5a6b37e5b7f064ea27261a6c5776ed082e16b24a" gracePeriod=30 Oct 30 01:00:02.880190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.880160 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c"] Oct 30 01:00:02.884186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.884156 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" containerID="cri-o://6052c86a9256d846fbf72501069a36ef5073d9395b3cf168fb2bce94f575d9f0" gracePeriod=30 Oct 30 01:00:02.886385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.886361 4905 generic.go:356] "Generic (PLEG): container finished" podID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerID="5c9d17cd6806946704771a993bb5f71b732e8b696e3c5c4fa7af74788730da2d" exitCode=60 Oct 30 01:00:02.886579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.886455 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerDied","Data":"5c9d17cd6806946704771a993bb5f71b732e8b696e3c5c4fa7af74788730da2d"} Oct 30 01:00:02.886919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.886901 4905 scope.go:122] "RemoveContainer" containerID="5c9d17cd6806946704771a993bb5f71b732e8b696e3c5c4fa7af74788730da2d" Oct 30 01:00:02.897333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.897311 4905 generic.go:356] "Generic (PLEG): container finished" podID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerID="904fc665dd9c81ef4c4b5030fb49ee166b399e3976bb615d0a16e66451d8f1f7" exitCode=60 Oct 30 01:00:02.897539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.897499 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerDied","Data":"904fc665dd9c81ef4c4b5030fb49ee166b399e3976bb615d0a16e66451d8f1f7"} Oct 30 01:00:02.897913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.897900 4905 scope.go:122] "RemoveContainer" containerID="904fc665dd9c81ef4c4b5030fb49ee166b399e3976bb615d0a16e66451d8f1f7" Oct 30 01:00:02.900440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.900410 4905 generic.go:356] "Generic (PLEG): container finished" podID="c1ad6109-5703-409d-8441-798c75dae1f8" containerID="18ed400e23ed551b6f45056a9be5bdd9dad5f22489726d74a6104dd71c178b42" exitCode=60 Oct 30 01:00:02.900587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.900567 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerDied","Data":"18ed400e23ed551b6f45056a9be5bdd9dad5f22489726d74a6104dd71c178b42"} Oct 30 01:00:02.900924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.900906 4905 scope.go:122] "RemoveContainer" containerID="18ed400e23ed551b6f45056a9be5bdd9dad5f22489726d74a6104dd71c178b42" Oct 30 01:00:02.913014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.912989 4905 generic.go:356] "Generic (PLEG): container finished" podID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerID="f43a7f33fe859e365dc457f6b73acda36e01e576ad4de7027032f6cb9968455d" exitCode=60 Oct 30 01:00:02.913217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.913185 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" event={"ID":"1397a8c5-495a-4b7e-a747-e9220a1e0869","Type":"ContainerDied","Data":"f43a7f33fe859e365dc457f6b73acda36e01e576ad4de7027032f6cb9968455d"} Oct 30 01:00:02.913711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.913692 4905 scope.go:122] "RemoveContainer" containerID="f43a7f33fe859e365dc457f6b73acda36e01e576ad4de7027032f6cb9968455d" Oct 30 01:00:02.920605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.918689 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" event={"ID":"5d0aa9d6-0277-4935-87c5-40538906eecf","Type":"ContainerDied","Data":"f45c7c28a2913924ac6595a37a9e86929fc17f3e46316d29b3a198ab7eca5ea8"} Oct 30 01:00:02.920605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.918731 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f45c7c28a2913924ac6595a37a9e86929fc17f3e46316d29b3a198ab7eca5ea8" Oct 30 01:00:02.920605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.918807 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw" Oct 30 01:00:02.943204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.942313 4905 generic.go:356] "Generic (PLEG): container finished" podID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerID="a98cabaf2034b027a931c1b2ce965f4a1f43a3de1eaf5349a69caa4dc9a0611c" exitCode=60 Oct 30 01:00:02.943204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.942498 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerDied","Data":"a98cabaf2034b027a931c1b2ce965f4a1f43a3de1eaf5349a69caa4dc9a0611c"} Oct 30 01:00:02.943436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.943404 4905 scope.go:122] "RemoveContainer" containerID="a98cabaf2034b027a931c1b2ce965f4a1f43a3de1eaf5349a69caa4dc9a0611c" Oct 30 01:00:02.953871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.953844 4905 generic.go:356] "Generic (PLEG): container finished" podID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerID="9f322422a0fcccf2a1d27a69eb0f988c8163802ba35f423450b7576dc9449d0d" exitCode=60 Oct 30 01:00:02.954066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.954038 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerDied","Data":"9f322422a0fcccf2a1d27a69eb0f988c8163802ba35f423450b7576dc9449d0d"} Oct 30 01:00:02.954767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:02.954745 4905 scope.go:122] "RemoveContainer" containerID="9f322422a0fcccf2a1d27a69eb0f988c8163802ba35f423450b7576dc9449d0d" Oct 30 01:00:03.157990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.157958 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn"] Oct 30 01:00:03.159323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.159296 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5d0aa9d6-0277-4935-87c5-40538906eecf" containerName="revoke-keys" Oct 30 01:00:03.159474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.159460 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0aa9d6-0277-4935-87c5-40538906eecf" containerName="revoke-keys" Oct 30 01:00:03.159842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.159823 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0aa9d6-0277-4935-87c5-40538906eecf" containerName="revoke-keys" Oct 30 01:00:03.174300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.173574 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.201800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.201772 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-869c367e-openshift-default-dockercfg-b9qvn\"" Oct 30 01:00:03.204328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.204299 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkrh\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-kube-api-access-2bkrh\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.204416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.204400 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-data\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.204469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.204454 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-certs\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.205246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.204751 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-istio-token\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.205246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.205071 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-envoy\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.205246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.205098 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b8803c-bbf8-434b-91de-024c924643cc-istiod-ca-cert\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.205246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.205193 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-credential-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.206022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.205322 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b8803c-bbf8-434b-91de-024c924643cc-istio-podinfo\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.206022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.205442 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.211208 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.211182 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn"] Oct 30 01:00:03.386809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.386491 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-istio-token\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.386946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.386920 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-envoy\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.387804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.387784 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b8803c-bbf8-434b-91de-024c924643cc-istiod-ca-cert\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.387958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.387943 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-credential-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.388122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.388110 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b8803c-bbf8-434b-91de-024c924643cc-istio-podinfo\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.388261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.388250 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.388355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.388344 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkrh\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-kube-api-access-2bkrh\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.388499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.388486 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-data\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.388639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.388624 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-certs\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.401609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.399452 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-certs\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.401609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.397781 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-credential-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.401609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.398295 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.401609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.398541 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-data\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.401609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.389337 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b8803c-bbf8-434b-91de-024c924643cc-istiod-ca-cert\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.436551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.421175 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b8803c-bbf8-434b-91de-024c924643cc-istio-podinfo\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.441387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.441355 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-istio-token\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.450872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.450841 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-envoy\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.463122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.463060 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkrh\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-kube-api-access-2bkrh\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-mljpn\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.512861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.512821 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:03.977134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.977034 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerStarted","Data":"ae0d625cdd54b79d9c1d205c113d77dc26e9e506ddd712f6c43d489ac2773441"} Oct 30 01:00:03.979318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.979293 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerStarted","Data":"c7efb5b218b9f80290131c3de66508a8dbca83f88c3b14dd4dbc6e5acb046c5e"} Oct 30 01:00:03.981779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.981757 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerStarted","Data":"ce4c57e5d6d894a081c442977f198b3ccce7fb5e928d632f2fcd331ba2af2340"} Oct 30 01:00:03.983864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.983843 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" event={"ID":"1397a8c5-495a-4b7e-a747-e9220a1e0869","Type":"ContainerStarted","Data":"97a32ab7d0467dd6d64e68ecb561edf276f70f1e0ca8e50cede7e5710b106c0f"} Oct 30 01:00:03.985821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.985799 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerStarted","Data":"8d99ac0aa5172d1c17454182047d856a45cca8cc047792fc037361fbdda7d2f6"} Oct 30 01:00:03.987647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:03.987626 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerStarted","Data":"b502ba83d3544453d287b2479795b24388233f7ac31efe7a2a926dc1fe291cc5"} Oct 30 01:00:04.118609 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:04.118569 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b8803c_bbf8_434b_91de_024c924643cc.slice/crio-939b3ff2391e837a418c8300c0a43b17e09004c706d0a19ab3fa963a13443787 WatchSource:0}: Error finding container 939b3ff2391e837a418c8300c0a43b17e09004c706d0a19ab3fa963a13443787: Status 404 returned error can't find the container with id 939b3ff2391e837a418c8300c0a43b17e09004c706d0a19ab3fa963a13443787 Oct 30 01:00:04.121405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.121374 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:04.121492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.121437 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:04.121492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.121450 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn"] Oct 30 01:00:04.121492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.121466 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:04.998569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.998535 4905 generic.go:356] "Generic (PLEG): container finished" podID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerID="8d99ac0aa5172d1c17454182047d856a45cca8cc047792fc037361fbdda7d2f6" exitCode=60 Oct 30 01:00:04.999076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.998611 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerDied","Data":"8d99ac0aa5172d1c17454182047d856a45cca8cc047792fc037361fbdda7d2f6"} Oct 30 01:00:04.999076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.998672 4905 scope.go:122] "RemoveContainer" containerID="a98cabaf2034b027a931c1b2ce965f4a1f43a3de1eaf5349a69caa4dc9a0611c" Oct 30 01:00:04.999076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:04.998993 4905 scope.go:122] "RemoveContainer" containerID="8d99ac0aa5172d1c17454182047d856a45cca8cc047792fc037361fbdda7d2f6" Oct 30 01:00:04.999409 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:04.999383 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv_odh-ai-gateway-infra(7e968a20-13f7-4bd2-8a94-ffa4d22dccb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" Oct 30 01:00:05.000809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.000788 4905 generic.go:356] "Generic (PLEG): container finished" podID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerID="b502ba83d3544453d287b2479795b24388233f7ac31efe7a2a926dc1fe291cc5" exitCode=60 Oct 30 01:00:05.000927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.000868 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerDied","Data":"b502ba83d3544453d287b2479795b24388233f7ac31efe7a2a926dc1fe291cc5"} Oct 30 01:00:05.001099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.001084 4905 scope.go:122] "RemoveContainer" containerID="b502ba83d3544453d287b2479795b24388233f7ac31efe7a2a926dc1fe291cc5" Oct 30 01:00:05.001448 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:05.001430 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr_odh-ai-gateway-infra(455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" Oct 30 01:00:05.002829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.002810 4905 generic.go:356] "Generic (PLEG): container finished" podID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerID="ae0d625cdd54b79d9c1d205c113d77dc26e9e506ddd712f6c43d489ac2773441" exitCode=60 Oct 30 01:00:05.003073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.003019 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerDied","Data":"ae0d625cdd54b79d9c1d205c113d77dc26e9e506ddd712f6c43d489ac2773441"} Oct 30 01:00:05.003193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.003169 4905 scope.go:122] "RemoveContainer" containerID="ae0d625cdd54b79d9c1d205c113d77dc26e9e506ddd712f6c43d489ac2773441" Oct 30 01:00:05.003481 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:05.003461 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-mdd8v_odh-ai-gateway-infra(47adc5ad-fcb5-4f50-864a-e0cd51b95bf7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" Oct 30 01:00:05.005277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.005256 4905 generic.go:356] "Generic (PLEG): container finished" podID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerID="c7efb5b218b9f80290131c3de66508a8dbca83f88c3b14dd4dbc6e5acb046c5e" exitCode=60 Oct 30 01:00:05.005382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.005288 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerDied","Data":"c7efb5b218b9f80290131c3de66508a8dbca83f88c3b14dd4dbc6e5acb046c5e"} Oct 30 01:00:05.005605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.005587 4905 scope.go:122] "RemoveContainer" containerID="c7efb5b218b9f80290131c3de66508a8dbca83f88c3b14dd4dbc6e5acb046c5e" Oct 30 01:00:05.005964 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:05.005940 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq_odh-ai-gateway-infra(2eb2b8df-479a-417d-8105-035cb8a422a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" Oct 30 01:00:05.008245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.008225 4905 generic.go:356] "Generic (PLEG): container finished" podID="c1ad6109-5703-409d-8441-798c75dae1f8" containerID="ce4c57e5d6d894a081c442977f198b3ccce7fb5e928d632f2fcd331ba2af2340" exitCode=60 Oct 30 01:00:05.008328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.008293 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerDied","Data":"ce4c57e5d6d894a081c442977f198b3ccce7fb5e928d632f2fcd331ba2af2340"} Oct 30 01:00:05.008506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.008492 4905 scope.go:122] "RemoveContainer" containerID="ce4c57e5d6d894a081c442977f198b3ccce7fb5e928d632f2fcd331ba2af2340" Oct 30 01:00:05.008772 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:05.008754 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r_odh-ai-gateway-infra(c1ad6109-5703-409d-8441-798c75dae1f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" Oct 30 01:00:05.010351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.010316 4905 generic.go:356] "Generic (PLEG): container finished" podID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerID="97a32ab7d0467dd6d64e68ecb561edf276f70f1e0ca8e50cede7e5710b106c0f" exitCode=60 Oct 30 01:00:05.010439 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.010390 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" event={"ID":"1397a8c5-495a-4b7e-a747-e9220a1e0869","Type":"ContainerDied","Data":"97a32ab7d0467dd6d64e68ecb561edf276f70f1e0ca8e50cede7e5710b106c0f"} Oct 30 01:00:05.010676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.010655 4905 scope.go:122] "RemoveContainer" containerID="97a32ab7d0467dd6d64e68ecb561edf276f70f1e0ca8e50cede7e5710b106c0f" Oct 30 01:00:05.010925 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:05.010901 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45_odh-ai-gateway-infra(1397a8c5-495a-4b7e-a747-e9220a1e0869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" Oct 30 01:00:05.012219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.012196 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" event={"ID":"34b8803c-bbf8-434b-91de-024c924643cc","Type":"ContainerStarted","Data":"af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d"} Oct 30 01:00:05.012313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.012239 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" event={"ID":"34b8803c-bbf8-434b-91de-024c924643cc","Type":"ContainerStarted","Data":"939b3ff2391e837a418c8300c0a43b17e09004c706d0a19ab3fa963a13443787"} Oct 30 01:00:05.014584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.014561 4905 scope.go:122] "RemoveContainer" containerID="9f322422a0fcccf2a1d27a69eb0f988c8163802ba35f423450b7576dc9449d0d" Oct 30 01:00:05.028693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.028676 4905 scope.go:122] "RemoveContainer" containerID="5c9d17cd6806946704771a993bb5f71b732e8b696e3c5c4fa7af74788730da2d" Oct 30 01:00:05.043512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.043489 4905 scope.go:122] "RemoveContainer" containerID="904fc665dd9c81ef4c4b5030fb49ee166b399e3976bb615d0a16e66451d8f1f7" Oct 30 01:00:05.058053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.058032 4905 scope.go:122] "RemoveContainer" containerID="18ed400e23ed551b6f45056a9be5bdd9dad5f22489726d74a6104dd71c178b42" Oct 30 01:00:05.073086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.073067 4905 scope.go:122] "RemoveContainer" containerID="f43a7f33fe859e365dc457f6b73acda36e01e576ad4de7027032f6cb9968455d" Oct 30 01:00:05.280636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.280584 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podStartSLOduration=2.280569969 podStartE2EDuration="2.280569969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.275053874 +0000 UTC m=+1290.218227069" watchObservedRunningTime="2026-10-30 01:00:05.280569969 +0000 UTC m=+1290.223743160" Oct 30 01:00:05.521545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.519882 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:05.543595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:05.542900 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:06.237697 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.237671 4905 scope.go:122] "RemoveContainer" containerID="8d99ac0aa5172d1c17454182047d856a45cca8cc047792fc037361fbdda7d2f6" Oct 30 01:00:06.238856 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:06.238820 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv_odh-ai-gateway-infra(7e968a20-13f7-4bd2-8a94-ffa4d22dccb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" Oct 30 01:00:06.243373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.243353 4905 scope.go:122] "RemoveContainer" containerID="b502ba83d3544453d287b2479795b24388233f7ac31efe7a2a926dc1fe291cc5" Oct 30 01:00:06.245117 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:06.244092 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr_odh-ai-gateway-infra(455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" Oct 30 01:00:06.247102 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.247080 4905 scope.go:122] "RemoveContainer" containerID="ae0d625cdd54b79d9c1d205c113d77dc26e9e506ddd712f6c43d489ac2773441" Oct 30 01:00:06.247645 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:06.247622 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-mdd8v_odh-ai-gateway-infra(47adc5ad-fcb5-4f50-864a-e0cd51b95bf7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" Oct 30 01:00:06.250010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.249992 4905 scope.go:122] "RemoveContainer" containerID="c7efb5b218b9f80290131c3de66508a8dbca83f88c3b14dd4dbc6e5acb046c5e" Oct 30 01:00:06.250287 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:06.250268 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq_odh-ai-gateway-infra(2eb2b8df-479a-417d-8105-035cb8a422a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" Oct 30 01:00:06.252656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.252130 4905 scope.go:122] "RemoveContainer" containerID="ce4c57e5d6d894a081c442977f198b3ccce7fb5e928d632f2fcd331ba2af2340" Oct 30 01:00:06.252656 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:06.252421 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r_odh-ai-gateway-infra(c1ad6109-5703-409d-8441-798c75dae1f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" Oct 30 01:00:06.254635 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.254617 4905 scope.go:122] "RemoveContainer" containerID="97a32ab7d0467dd6d64e68ecb561edf276f70f1e0ca8e50cede7e5710b106c0f" Oct 30 01:00:06.254765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.254746 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:06.254862 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:06.254844 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45_odh-ai-gateway-infra(1397a8c5-495a-4b7e-a747-e9220a1e0869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" Oct 30 01:00:06.255691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:06.255674 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:00:08.362845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.358947 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 01:00:08.367342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.366728 4905 generic.go:356] "Generic (PLEG): container finished" podID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerID="f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26" exitCode=0 Oct 30 01:00:08.367342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.366936 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" event={"ID":"575fa231-7d17-4522-bd3e-cc8bebe2e964","Type":"ContainerDied","Data":"f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26"} Oct 30 01:00:08.367342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.366989 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" event={"ID":"575fa231-7d17-4522-bd3e-cc8bebe2e964","Type":"ContainerDied","Data":"fc92bb083164210303ccf6992ce517a01e4aa6721d9f4e941f0cf69dffc9fe39"} Oct 30 01:00:08.367342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.367018 4905 scope.go:122] "RemoveContainer" containerID="f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26" Oct 30 01:00:08.367342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.367166 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm" Oct 30 01:00:08.372727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.372700 4905 generic.go:356] "Generic (PLEG): container finished" podID="25db140f-01b8-496e-89c5-c26b749ba26f" containerID="bc77cd748485055be227c44c5a6b37e5b7f064ea27261a6c5776ed082e16b24a" exitCode=0 Oct 30 01:00:08.372862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.372842 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" event={"ID":"25db140f-01b8-496e-89c5-c26b749ba26f","Type":"ContainerDied","Data":"bc77cd748485055be227c44c5a6b37e5b7f064ea27261a6c5776ed082e16b24a"} Oct 30 01:00:08.374849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.374649 4905 generic.go:356] "Generic (PLEG): container finished" podID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerID="d8d26e6acb18df183a97d45bd4e2120b75334e27342d6778fac363b8aca5ebbf" exitCode=0 Oct 30 01:00:08.374849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.374685 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" event={"ID":"4ef41398-006e-4f20-aa8b-1a8dd9d4092e","Type":"ContainerDied","Data":"d8d26e6acb18df183a97d45bd4e2120b75334e27342d6778fac363b8aca5ebbf"} Oct 30 01:00:08.376485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.376466 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 01:00:08.379383 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.379364 4905 generic.go:356] "Generic (PLEG): container finished" podID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerID="6052c86a9256d846fbf72501069a36ef5073d9395b3cf168fb2bce94f575d9f0" exitCode=0 Oct 30 01:00:08.380440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.380418 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" event={"ID":"774bdea2-f6dc-4d88-9cc9-91af7d21ae12","Type":"ContainerDied","Data":"6052c86a9256d846fbf72501069a36ef5073d9395b3cf168fb2bce94f575d9f0"} Oct 30 01:00:08.474131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.474043 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-certs\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.474415 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.474387 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-podinfo\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.474638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.474607 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istiod-ca-cert\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.474790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.474772 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.474914 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-envoy\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.474968 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-data\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475044 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-credential-socket\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475132 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575fa231-7d17-4522-bd3e-cc8bebe2e964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575fa231-7d17-4522-bd3e-cc8bebe2e964-istiod-ca-cert\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475211 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-kube-api-access-2zlh4\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-kube-api-access-2zlh4\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475309 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-socket\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475356 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-socket\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475391 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-kube-api-access-pkcd2\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-kube-api-access-pkcd2\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475446 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-token\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-token\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475552 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-certs\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475593 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-token\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-token\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475635 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-data\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475706 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-podinfo\") pod \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\" (UID: \"4ef41398-006e-4f20-aa8b-1a8dd9d4092e\") " Oct 30 01:00:08.476970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.475902 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-credential-socket\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.486486 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.486454 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-certs" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.487873 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.487832 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-credential-socket" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.488256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.488234 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-certs" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.509004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.502650 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-data" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.538631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.519804 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-credential-socket" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.538631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.520473 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istiod-ca-cert" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:08.538631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.520780 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-socket" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.538631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.521080 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575fa231-7d17-4522-bd3e-cc8bebe2e964-istiod-ca-cert" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:08.566940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.566899 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-socket" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.580711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.580683 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-kube-api-access-pkcd2" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "kube-api-access-pkcd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.581340 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.581318 4905 scope.go:122] "RemoveContainer" containerID="f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26" Oct 30 01:00:08.581960 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.581934 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.582110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582090 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\") pod \"575fa231-7d17-4522-bd3e-cc8bebe2e964\" (UID: \"575fa231-7d17-4522-bd3e-cc8bebe2e964\") " Oct 30 01:00:08.582269 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:08.582253 4905 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/575fa231-7d17-4522-bd3e-cc8bebe2e964/volumes/kubernetes.io~empty-dir/istio-envoy Oct 30 01:00:08.582322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582271 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.582371 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:08.582316 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26\": container with ID starting with f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26 not found: ID does not exist" containerID="f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26" Oct 30 01:00:08.582371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582344 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26"} err="failed to get container status \"f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26\": rpc error: code = NotFound desc = could not find container \"f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26\": container with ID starting with f127d2ce8165a47f058b550de41f385686c9033ccc4133433536a5da2d6d6c26 not found: ID does not exist" Oct 30 01:00:08.582443 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582374 4905 scope.go:122] "RemoveContainer" containerID="683c692b6c7bf3f54b3992c1c282885468c7f6f5b59c6b08bed03dfcbdb75efa" Oct 30 01:00:08.582495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582481 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-data" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.582945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582910 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582947 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582957 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkcd2\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-kube-api-access-pkcd2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582967 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582975 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582983 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.582990 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.583003 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.583016 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.583023 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.583030 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.583038 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575fa231-7d17-4522-bd3e-cc8bebe2e964-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.583721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.583626 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-podinfo" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:08.587980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.587953 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-podinfo" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:08.588083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.588024 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-token" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.588083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.588053 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-token" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.590862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.590836 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-kube-api-access-2zlh4" pod "575fa231-7d17-4522-bd3e-cc8bebe2e964" (UID: "575fa231-7d17-4522-bd3e-cc8bebe2e964"). InnerVolumeSpecName "kube-api-access-2zlh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.591003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.590979 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-envoy" pod "4ef41398-006e-4f20-aa8b-1a8dd9d4092e" (UID: "4ef41398-006e-4f20-aa8b-1a8dd9d4092e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.599063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.599042 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 01:00:08.602683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.602663 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 01:00:08.683501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683470 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-credential-socket\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.683700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683512 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-envoy\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.683700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683589 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-kube-api-access-g4p76\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-kube-api-access-g4p76\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.683700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683695 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-socket\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.683863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683727 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-kube-api-access-7wrg6\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-kube-api-access-7wrg6\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.683863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683796 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25db140f-01b8-496e-89c5-c26b749ba26f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25db140f-01b8-496e-89c5-c26b749ba26f-istio-podinfo\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.683863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683831 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-data\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.684018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683875 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-istio-token\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-istio-token\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.684018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683929 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istiod-ca-cert\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683970 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-envoy\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.683987 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-certs\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684013 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-data\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684059 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25db140f-01b8-496e-89c5-c26b749ba26f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25db140f-01b8-496e-89c5-c26b749ba26f-istiod-ca-cert\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.684329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684108 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-certs\") pod \"25db140f-01b8-496e-89c5-c26b749ba26f\" (UID: \"25db140f-01b8-496e-89c5-c26b749ba26f\") " Oct 30 01:00:08.684329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684142 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-token\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-token\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684214 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-credential-socket\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684241 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-podinfo\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684261 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-socket\") pod \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\" (UID: \"774bdea2-f6dc-4d88-9cc9-91af7d21ae12\") " Oct 30 01:00:08.684962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684940 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlh4\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-kube-api-access-2zlh4\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.685074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684967 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.685074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684980 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.685074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.684992 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.685074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685003 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575fa231-7d17-4522-bd3e-cc8bebe2e964-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.685074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685015 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef41398-006e-4f20-aa8b-1a8dd9d4092e-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.685342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685092 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istiod-ca-cert" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:08.685342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685267 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-socket" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.685452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685357 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25db140f-01b8-496e-89c5-c26b749ba26f-istiod-ca-cert" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:08.685502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685486 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-certs" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.685669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685212 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-socket" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.685669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685646 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-data" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.685844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.685729 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-credential-socket" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.686142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.686114 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-credential-socket" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.686335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.686315 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-certs" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.686703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.686679 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-data" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.689601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.689431 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-envoy" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.689601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.689561 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-token" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.690889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.690870 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-kube-api-access-7wrg6" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "kube-api-access-7wrg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.691184 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.691155 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25db140f-01b8-496e-89c5-c26b749ba26f-istio-podinfo" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:08.691395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.691359 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-envoy" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:08.691506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.691441 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-istio-token" pod "25db140f-01b8-496e-89c5-c26b749ba26f" (UID: "25db140f-01b8-496e-89c5-c26b749ba26f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.691506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.691467 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-kube-api-access-g4p76" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "kube-api-access-g4p76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:08.692448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.692425 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-podinfo" pod "774bdea2-f6dc-4d88-9cc9-91af7d21ae12" (UID: "774bdea2-f6dc-4d88-9cc9-91af7d21ae12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:08.748602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.748572 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm"] Oct 30 01:00:08.754639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.754612 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-g5ltm"] Oct 30 01:00:08.786644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786614 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786638 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786651 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786663 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786674 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786684 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786695 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25db140f-01b8-496e-89c5-c26b749ba26f-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786706 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786716 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786727 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786741 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786754 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786770 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786781 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786794 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4p76\" (UniqueName: \"kubernetes.io/projected/774bdea2-f6dc-4d88-9cc9-91af7d21ae12-kube-api-access-g4p76\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786805 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25db140f-01b8-496e-89c5-c26b749ba26f-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786819 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrg6\" (UniqueName: \"kubernetes.io/projected/25db140f-01b8-496e-89c5-c26b749ba26f-kube-api-access-7wrg6\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:08.786854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:08.786832 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25db140f-01b8-496e-89c5-c26b749ba26f-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:09.193434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.193362 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" path="/var/lib/kubelet/pods/575fa231-7d17-4522-bd3e-cc8bebe2e964/volumes" Oct 30 01:00:09.392792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.392761 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" Oct 30 01:00:09.393254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.392746 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds" event={"ID":"25db140f-01b8-496e-89c5-c26b749ba26f","Type":"ContainerDied","Data":"cd11744c6d3f80649fdcb7e20778429749eea920fa6bb84cf6b3b6eca3c4be70"} Oct 30 01:00:09.393254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.393007 4905 scope.go:122] "RemoveContainer" containerID="bc77cd748485055be227c44c5a6b37e5b7f064ea27261a6c5776ed082e16b24a" Oct 30 01:00:09.395334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.395316 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" Oct 30 01:00:09.395481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.395328 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4" event={"ID":"4ef41398-006e-4f20-aa8b-1a8dd9d4092e","Type":"ContainerDied","Data":"44eb38626d73189b6a2759b3f70e015c05ed5b401efbd4841a26d4d7324845fc"} Oct 30 01:00:09.397616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.397348 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" event={"ID":"774bdea2-f6dc-4d88-9cc9-91af7d21ae12","Type":"ContainerDied","Data":"f1a4f91522d36238290ddf378890eea3b7295f5eb8ce93e36430e9b7b653dc9b"} Oct 30 01:00:09.397616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.397403 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c" Oct 30 01:00:09.406866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.406845 4905 scope.go:122] "RemoveContainer" containerID="d8d26e6acb18df183a97d45bd4e2120b75334e27342d6778fac363b8aca5ebbf" Oct 30 01:00:09.420155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.420133 4905 scope.go:122] "RemoveContainer" containerID="6052c86a9256d846fbf72501069a36ef5073d9395b3cf168fb2bce94f575d9f0" Oct 30 01:00:09.460782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.460712 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds"] Oct 30 01:00:09.470886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.470861 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ts9ds"] Oct 30 01:00:09.535379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.535349 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c"] Oct 30 01:00:09.540315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.540287 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6w6c"] Oct 30 01:00:09.575484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.575453 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4"] Oct 30 01:00:09.580491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.580468 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8zkx4"] Oct 30 01:00:09.664677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.664648 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v"] Oct 30 01:00:09.665339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665318 4905 cpu_manager.go:414] "containerMap: removing container" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" Oct 30 01:00:09.665339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665335 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665355 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665359 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665373 4905 cpu_manager.go:414] "containerMap: removing container" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665378 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665386 4905 cpu_manager.go:414] "containerMap: removing container" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665389 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665409 4905 cpu_manager.go:414] "containerMap: removing container" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" Oct 30 01:00:09.665483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665413 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" Oct 30 01:00:09.665745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665538 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" Oct 30 01:00:09.665745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665552 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" containerName="istio-proxy" Oct 30 01:00:09.665745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665560 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" containerName="istio-proxy" Oct 30 01:00:09.665745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665568 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" containerName="istio-proxy" Oct 30 01:00:09.665745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.665575 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="575fa231-7d17-4522-bd3e-cc8bebe2e964" containerName="istio-proxy" Oct 30 01:00:09.671296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.671279 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:09.688077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.688052 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-869c367e\"" Oct 30 01:00:09.689877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.689858 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v"] Oct 30 01:00:09.700209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.700169 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbm2q\" (UniqueName: \"kubernetes.io/projected/985442ab-ef94-4ab5-8301-720ba216d8b4-kube-api-access-kbm2q\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:09.700318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.700267 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985442ab-ef94-4ab5-8301-720ba216d8b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:09.826575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.823015 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-869c367e-dockercfg-zzbgp\"" Oct 30 01:00:09.901564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.890147 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbm2q\" (UniqueName: \"kubernetes.io/projected/985442ab-ef94-4ab5-8301-720ba216d8b4-kube-api-access-kbm2q\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:09.923723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.923677 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985442ab-ef94-4ab5-8301-720ba216d8b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:09.937282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.924946 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985442ab-ef94-4ab5-8301-720ba216d8b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:10.002431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:09.998152 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbm2q\" (UniqueName: \"kubernetes.io/projected/985442ab-ef94-4ab5-8301-720ba216d8b4-kube-api-access-kbm2q\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:10.018213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.018181 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs"] Oct 30 01:00:10.023910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.023321 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.044585 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.041850 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs"] Oct 30 01:00:10.128637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.128544 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06939e6-3c32-4a14-9e73-579e77c04808-plugins-config-volume\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.129060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.129040 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthp2\" (UniqueName: \"kubernetes.io/projected/d06939e6-3c32-4a14-9e73-579e77c04808-kube-api-access-xthp2\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.230802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.230761 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthp2\" (UniqueName: \"kubernetes.io/projected/d06939e6-3c32-4a14-9e73-579e77c04808-kube-api-access-xthp2\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.231076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.230989 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06939e6-3c32-4a14-9e73-579e77c04808-plugins-config-volume\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.231870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.231843 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06939e6-3c32-4a14-9e73-579e77c04808-plugins-config-volume\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.251424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.251384 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthp2\" (UniqueName: \"kubernetes.io/projected/d06939e6-3c32-4a14-9e73-579e77c04808-kube-api-access-xthp2\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.287977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.287949 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:10.359013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.358730 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:10.776655 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:10.776619 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06939e6_3c32_4a14_9e73_579e77c04808.slice/crio-486e15ec81fda8961d14a4562ff162893867153121b2d4b859e88b29dbeda45d WatchSource:0}: Error finding container 486e15ec81fda8961d14a4562ff162893867153121b2d4b859e88b29dbeda45d: Status 404 returned error can't find the container with id 486e15ec81fda8961d14a4562ff162893867153121b2d4b859e88b29dbeda45d Oct 30 01:00:10.778732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.778705 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs"] Oct 30 01:00:10.793959 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:10.793934 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985442ab_ef94_4ab5_8301_720ba216d8b4.slice/crio-901ca2527e426334c17dc9f61f6ba3bbef08f7eaa235a3adf569b00d29070a36 WatchSource:0}: Error finding container 901ca2527e426334c17dc9f61f6ba3bbef08f7eaa235a3adf569b00d29070a36: Status 404 returned error can't find the container with id 901ca2527e426334c17dc9f61f6ba3bbef08f7eaa235a3adf569b00d29070a36 Oct 30 01:00:10.797343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:10.797314 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v"] Oct 30 01:00:11.193109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.193036 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25db140f-01b8-496e-89c5-c26b749ba26f" path="/var/lib/kubelet/pods/25db140f-01b8-496e-89c5-c26b749ba26f/volumes" Oct 30 01:00:11.193534 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.193502 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef41398-006e-4f20-aa8b-1a8dd9d4092e" path="/var/lib/kubelet/pods/4ef41398-006e-4f20-aa8b-1a8dd9d4092e/volumes" Oct 30 01:00:11.193952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.193938 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774bdea2-f6dc-4d88-9cc9-91af7d21ae12" path="/var/lib/kubelet/pods/774bdea2-f6dc-4d88-9cc9-91af7d21ae12/volumes" Oct 30 01:00:11.438477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.438441 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk"] Oct 30 01:00:11.445973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.445912 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.489899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.489864 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk"] Oct 30 01:00:11.490950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.490923 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4b973b-openshift-default-dockercfg-6znzx\"" Oct 30 01:00:11.492151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.492123 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-credential-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.492378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.492317 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7s8\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-kube-api-access-th7s8\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.492590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.492571 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.492778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.492761 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-token\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.492957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.492936 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-podinfo\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.493079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.493063 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-envoy\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.493246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.493230 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-certs\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.493409 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.493392 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-data\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.493575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.493559 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.624861 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-credential-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.624928 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7s8\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-kube-api-access-th7s8\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.624989 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.625074 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-token\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.625159 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-podinfo\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.625187 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-envoy\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.625253 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-certs\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.625317 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-data\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.625362 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.626281 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.626572 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-credential-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.651545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.627180 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.652882 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.652848 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-certs\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.653728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.653668 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-data\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.702452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.702375 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-envoy\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.717254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.717110 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" event={"ID":"985442ab-ef94-4ab5-8301-720ba216d8b4","Type":"ContainerStarted","Data":"dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2"} Oct 30 01:00:11.717254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.717179 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" event={"ID":"985442ab-ef94-4ab5-8301-720ba216d8b4","Type":"ContainerStarted","Data":"901ca2527e426334c17dc9f61f6ba3bbef08f7eaa235a3adf569b00d29070a36"} Oct 30 01:00:11.717254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.717217 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:11.720563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.720223 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" event={"ID":"d06939e6-3c32-4a14-9e73-579e77c04808","Type":"ContainerStarted","Data":"d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd"} Oct 30 01:00:11.720563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.720274 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" event={"ID":"d06939e6-3c32-4a14-9e73-579e77c04808","Type":"ContainerStarted","Data":"486e15ec81fda8961d14a4562ff162893867153121b2d4b859e88b29dbeda45d"} Oct 30 01:00:11.721733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.721665 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:11.722755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.722729 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-podinfo\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.735667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.727687 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-token\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.753545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.748164 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7s8\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-kube-api-access-th7s8\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.816775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.816743 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:11.817509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.817048 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" podStartSLOduration=2.817028988 podStartE2EDuration="2.817028988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:11.770247251 +0000 UTC m=+1296.713420445" watchObservedRunningTime="2026-10-30 01:00:11.817028988 +0000 UTC m=+1296.760202181" Oct 30 01:00:11.913117 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:11.905549 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" podStartSLOduration=2.905503057 podStartE2EDuration="2.905503057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:11.856405291 +0000 UTC m=+1296.799578483" watchObservedRunningTime="2026-10-30 01:00:11.905503057 +0000 UTC m=+1296.848676251" Oct 30 01:00:12.465907 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:12.463692 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:00:12.646236 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:12.646055 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e4567a_0444_4693_a2f0_c832eaf6e6ff.slice/crio-70437262e6274284276d912ab5474459258a3ba9edeffda97443c89855c2a7cd WatchSource:0}: Error finding container 70437262e6274284276d912ab5474459258a3ba9edeffda97443c89855c2a7cd: Status 404 returned error can't find the container with id 70437262e6274284276d912ab5474459258a3ba9edeffda97443c89855c2a7cd Oct 30 01:00:12.648706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:12.648556 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk"] Oct 30 01:00:12.650646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:12.650608 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:12.650778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:12.650755 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:12.650952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:12.650860 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:12.733137 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:12.733099 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" event={"ID":"90e4567a-0444-4693-a2f0-c832eaf6e6ff","Type":"ContainerStarted","Data":"70437262e6274284276d912ab5474459258a3ba9edeffda97443c89855c2a7cd"} Oct 30 01:00:12.778422 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:12.778380 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw"] Oct 30 01:00:12.784593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:12.784568 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-95b31f79-74xxw"] Oct 30 01:00:13.280474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:13.280438 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0aa9d6-0277-4935-87c5-40538906eecf" path="/var/lib/kubelet/pods/5d0aa9d6-0277-4935-87c5-40538906eecf/volumes" Oct 30 01:00:13.746412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:13.746376 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" event={"ID":"90e4567a-0444-4693-a2f0-c832eaf6e6ff","Type":"ContainerStarted","Data":"b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc"} Oct 30 01:00:13.817923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:13.817897 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:13.823337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:13.823315 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:13.823985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:13.823947 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podStartSLOduration=2.823933566 podStartE2EDuration="2.823933566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:13.80450427 +0000 UTC m=+1298.747677491" watchObservedRunningTime="2026-10-30 01:00:13.823933566 +0000 UTC m=+1298.767106784" Oct 30 01:00:14.759275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:14.759245 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:14.760433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:14.760404 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:00:15.189040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.188920 4905 scope.go:122] "RemoveContainer" containerID="97a32ab7d0467dd6d64e68ecb561edf276f70f1e0ca8e50cede7e5710b106c0f" Oct 30 01:00:15.189189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.189042 4905 scope.go:122] "RemoveContainer" containerID="b502ba83d3544453d287b2479795b24388233f7ac31efe7a2a926dc1fe291cc5" Oct 30 01:00:15.189189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.189173 4905 scope.go:122] "RemoveContainer" containerID="8d99ac0aa5172d1c17454182047d856a45cca8cc047792fc037361fbdda7d2f6" Oct 30 01:00:15.189886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.189512 4905 scope.go:122] "RemoveContainer" containerID="ae0d625cdd54b79d9c1d205c113d77dc26e9e506ddd712f6c43d489ac2773441" Oct 30 01:00:15.773155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.773123 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerStarted","Data":"1c7e8b726462272424a66abdd7280e313bed59436a79a419a741c61d5214b656"} Oct 30 01:00:15.775393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.775371 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerStarted","Data":"716d88d93a6bcb6d01018d6b8d8d517ee6ded7215e3d060f92f49163ddb46d0e"} Oct 30 01:00:15.777313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.777286 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerStarted","Data":"60b9156f053571f1599da079d5409629499bbd56aea2f4b0ab30915cbf8dd949"} Oct 30 01:00:15.779276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:15.779252 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" event={"ID":"1397a8c5-495a-4b7e-a747-e9220a1e0869","Type":"ContainerStarted","Data":"64cc9d0b0c64ea07351b70a1be7702d5e769e6f33532abe641d62c5f65c9cddc"} Oct 30 01:00:16.183959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.183924 4905 scope.go:122] "RemoveContainer" containerID="ce4c57e5d6d894a081c442977f198b3ccce7fb5e928d632f2fcd331ba2af2340" Oct 30 01:00:16.184097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.183998 4905 scope.go:122] "RemoveContainer" containerID="c7efb5b218b9f80290131c3de66508a8dbca83f88c3b14dd4dbc6e5acb046c5e" Oct 30 01:00:16.795774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.795736 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerStarted","Data":"0f19d7da30a829db549576ffa1f107a7d2653a281074e78e37f86e7c6a8d3a58"} Oct 30 01:00:16.798313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.798279 4905 generic.go:356] "Generic (PLEG): container finished" podID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerID="64cc9d0b0c64ea07351b70a1be7702d5e769e6f33532abe641d62c5f65c9cddc" exitCode=60 Oct 30 01:00:16.798430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.798343 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" event={"ID":"1397a8c5-495a-4b7e-a747-e9220a1e0869","Type":"ContainerDied","Data":"64cc9d0b0c64ea07351b70a1be7702d5e769e6f33532abe641d62c5f65c9cddc"} Oct 30 01:00:16.798430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.798415 4905 scope.go:122] "RemoveContainer" containerID="97a32ab7d0467dd6d64e68ecb561edf276f70f1e0ca8e50cede7e5710b106c0f" Oct 30 01:00:16.798783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.798764 4905 scope.go:122] "RemoveContainer" containerID="64cc9d0b0c64ea07351b70a1be7702d5e769e6f33532abe641d62c5f65c9cddc" Oct 30 01:00:16.799140 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:16.799111 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45_odh-ai-gateway-infra(1397a8c5-495a-4b7e-a747-e9220a1e0869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" Oct 30 01:00:16.800820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.800800 4905 generic.go:356] "Generic (PLEG): container finished" podID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerID="1c7e8b726462272424a66abdd7280e313bed59436a79a419a741c61d5214b656" exitCode=60 Oct 30 01:00:16.800925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.800871 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerDied","Data":"1c7e8b726462272424a66abdd7280e313bed59436a79a419a741c61d5214b656"} Oct 30 01:00:16.801188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.801169 4905 scope.go:122] "RemoveContainer" containerID="1c7e8b726462272424a66abdd7280e313bed59436a79a419a741c61d5214b656" Oct 30 01:00:16.801462 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:16.801439 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv_odh-ai-gateway-infra(7e968a20-13f7-4bd2-8a94-ffa4d22dccb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" Oct 30 01:00:16.802643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.802624 4905 generic.go:356] "Generic (PLEG): container finished" podID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerID="716d88d93a6bcb6d01018d6b8d8d517ee6ded7215e3d060f92f49163ddb46d0e" exitCode=60 Oct 30 01:00:16.802714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.802697 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerDied","Data":"716d88d93a6bcb6d01018d6b8d8d517ee6ded7215e3d060f92f49163ddb46d0e"} Oct 30 01:00:16.802976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.802956 4905 scope.go:122] "RemoveContainer" containerID="716d88d93a6bcb6d01018d6b8d8d517ee6ded7215e3d060f92f49163ddb46d0e" Oct 30 01:00:16.803235 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:16.803213 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr_odh-ai-gateway-infra(455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" Oct 30 01:00:16.804872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.804854 4905 generic.go:356] "Generic (PLEG): container finished" podID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerID="60b9156f053571f1599da079d5409629499bbd56aea2f4b0ab30915cbf8dd949" exitCode=60 Oct 30 01:00:16.804971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.804902 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerDied","Data":"60b9156f053571f1599da079d5409629499bbd56aea2f4b0ab30915cbf8dd949"} Oct 30 01:00:16.805230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.805212 4905 scope.go:122] "RemoveContainer" containerID="60b9156f053571f1599da079d5409629499bbd56aea2f4b0ab30915cbf8dd949" Oct 30 01:00:16.805534 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:16.805495 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-mdd8v_odh-ai-gateway-infra(47adc5ad-fcb5-4f50-864a-e0cd51b95bf7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" Oct 30 01:00:16.807202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.807179 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerStarted","Data":"96af3d0f8362b85ea7a5bf993cfa4cbdd1d7da91e02f63abb140890ce55f4b54"} Oct 30 01:00:16.813382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.813365 4905 scope.go:122] "RemoveContainer" containerID="8d99ac0aa5172d1c17454182047d856a45cca8cc047792fc037361fbdda7d2f6" Oct 30 01:00:16.829758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.829735 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv"] Oct 30 01:00:16.831138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.831124 4905 scope.go:122] "RemoveContainer" containerID="b502ba83d3544453d287b2479795b24388233f7ac31efe7a2a926dc1fe291cc5" Oct 30 01:00:16.852216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.852194 4905 scope.go:122] "RemoveContainer" containerID="ae0d625cdd54b79d9c1d205c113d77dc26e9e506ddd712f6c43d489ac2773441" Oct 30 01:00:16.870895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.870872 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v"] Oct 30 01:00:16.906507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.906486 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr"] Oct 30 01:00:16.948060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:16.948031 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45"] Oct 30 01:00:17.826689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.826653 4905 generic.go:356] "Generic (PLEG): container finished" podID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerID="96af3d0f8362b85ea7a5bf993cfa4cbdd1d7da91e02f63abb140890ce55f4b54" exitCode=60 Oct 30 01:00:17.827004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.826780 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerDied","Data":"96af3d0f8362b85ea7a5bf993cfa4cbdd1d7da91e02f63abb140890ce55f4b54"} Oct 30 01:00:17.827004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.826832 4905 scope.go:122] "RemoveContainer" containerID="c7efb5b218b9f80290131c3de66508a8dbca83f88c3b14dd4dbc6e5acb046c5e" Oct 30 01:00:17.827235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.827210 4905 scope.go:122] "RemoveContainer" containerID="96af3d0f8362b85ea7a5bf993cfa4cbdd1d7da91e02f63abb140890ce55f4b54" Oct 30 01:00:17.827635 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:17.827605 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq_odh-ai-gateway-infra(2eb2b8df-479a-417d-8105-035cb8a422a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" Oct 30 01:00:17.831793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.831739 4905 generic.go:356] "Generic (PLEG): container finished" podID="c1ad6109-5703-409d-8441-798c75dae1f8" containerID="0f19d7da30a829db549576ffa1f107a7d2653a281074e78e37f86e7c6a8d3a58" exitCode=60 Oct 30 01:00:17.832007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.831796 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerDied","Data":"0f19d7da30a829db549576ffa1f107a7d2653a281074e78e37f86e7c6a8d3a58"} Oct 30 01:00:17.832195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.832178 4905 scope.go:122] "RemoveContainer" containerID="0f19d7da30a829db549576ffa1f107a7d2653a281074e78e37f86e7c6a8d3a58" Oct 30 01:00:17.832636 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:17.832610 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r_odh-ai-gateway-infra(c1ad6109-5703-409d-8441-798c75dae1f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" Oct 30 01:00:17.849789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.849682 4905 scope.go:122] "RemoveContainer" containerID="ce4c57e5d6d894a081c442977f198b3ccce7fb5e928d632f2fcd331ba2af2340" Oct 30 01:00:17.870431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.870398 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r"] Oct 30 01:00:17.883615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.880099 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc"] Oct 30 01:00:17.888941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.888415 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:17.926591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.921320 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rxg2r\"" Oct 30 01:00:17.976541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.973604 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd"] Oct 30 01:00:17.991945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.991921 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:17.992589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.991821 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc"] Oct 30 01:00:17.993199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.993161 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7"] Oct 30 01:00:17.999441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.999429 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd"] Oct 30 01:00:18.001182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:17.999669 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.006480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006454 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.006716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006697 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.006765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006745 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.006831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006820 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.006869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006856 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.006906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006872 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.006906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006892 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.006966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.006949 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtkv\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-kube-api-access-xjtkv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.007257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.007242 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.011148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.011127 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7"] Oct 30 01:00:18.018211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.018196 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq"] Oct 30 01:00:18.108949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.108904 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.109361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.109196 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtkv\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-kube-api-access-xjtkv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.109361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.109245 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.109361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.109294 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.109361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.109322 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.109900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.109693 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.109900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.109781 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.109900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.109815 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110133 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110169 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110236 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr629\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-kube-api-access-lr629\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110275 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110356 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110392 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110417 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110440 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110494 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110591 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110619 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110679 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110715 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110751 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110806 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110831 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7q9\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-kube-api-access-4c7q9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110860 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110892 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.110917 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb9abe-dc1e-4805-9547-63c4606f6570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.112288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.111675 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.115186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.113721 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.116364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.115901 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.116674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.116653 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.117808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.116948 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.158183 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.158147 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67"] Oct 30 01:00:18.170707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.168282 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.170707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.169686 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtkv\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-kube-api-access-xjtkv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.170707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.170322 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.174198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.174176 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zz5pc\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.199859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.199819 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67"] Oct 30 01:00:18.200419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.200397 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:18.217274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217245 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.217434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217300 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.217434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217338 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.217434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217386 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7q9\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-kube-api-access-4c7q9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.217434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217422 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb9abe-dc1e-4805-9547-63c4606f6570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.217790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217485 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.217790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217555 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.217790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217609 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.217790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217691 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.217790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217727 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.218109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217802 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.218109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217832 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.218109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217898 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr629\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-kube-api-access-lr629\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.218109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.217937 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.218109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.218023 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.218109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.218050 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.218109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.218076 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.218606 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.218154 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.218664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.218616 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.219901 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.219486 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.221207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.220200 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.221207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.220948 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.223007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.222982 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4b973b\"" Oct 30 01:00:18.223274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.223253 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.225738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.223774 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.225738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.223795 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.225738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.224003 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.225738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.225699 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.231225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.227793 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.231225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.229814 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb9abe-dc1e-4805-9547-63c4606f6570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.231372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.231231 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.231880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.231596 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.242774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.242747 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.244218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.244187 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:18.255998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.255920 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7q9\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-kube-api-access-4c7q9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.267093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.267066 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:18.297434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.297398 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:18.311908 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.311384 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4b973b-dockercfg-r6q8w\"" Oct 30 01:00:18.337277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.323753 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-maas-api-service-ca\") pod \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " Oct 30 01:00:18.337277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.323958 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-kube-api-access-xmg6q\" (UniqueName: \"kubernetes.io/projected/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-kube-api-access-xmg6q\") pod \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " Oct 30 01:00:18.337277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.324076 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-maas-api-service-ca\") pod \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\" (UID: \"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8\") " Oct 30 01:00:18.337277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.324185 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-kube-api-access-wdzc5\" (UniqueName: \"kubernetes.io/projected/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-kube-api-access-wdzc5\") pod \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\" (UID: \"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7\") " Oct 30 01:00:18.355511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.353158 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9hp\" (UniqueName: \"kubernetes.io/projected/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-kube-api-access-6c9hp\") pod \"payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:18.355511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.353672 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:18.355511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.354813 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-maas-api-service-ca" pod "7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" (UID: "7e968a20-13f7-4bd2-8a94-ffa4d22dccb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:18.355810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.355554 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:18.400541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.381551 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-maas-api-service-ca" pod "455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" (UID: "455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:18.419806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.419767 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-kube-api-access-xmg6q" pod "7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" (UID: "7e968a20-13f7-4bd2-8a94-ffa4d22dccb8"). InnerVolumeSpecName "kube-api-access-xmg6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.470713 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9hp\" (UniqueName: \"kubernetes.io/projected/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-kube-api-access-6c9hp\") pod \"payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.667216 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.667711 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.667744 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmg6q\" (UniqueName: \"kubernetes.io/projected/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-kube-api-access-xmg6q\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.667758 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.521025 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr629\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-kube-api-access-lr629\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.509676 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vglwd\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.668675 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.510252 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-96bv7\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.676234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.668855 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-kube-api-access-wdzc5" pod "455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" (UID: "455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7"). InnerVolumeSpecName "kube-api-access-wdzc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.691781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.689420 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:18.702115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.702088 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9hp\" (UniqueName: \"kubernetes.io/projected/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-kube-api-access-6c9hp\") pod \"payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:18.708431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.708405 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:18.769295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.769256 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-maas-api-service-ca\") pod \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " Oct 30 01:00:18.769504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.769485 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-kube-api-access-4m7bg\" (UniqueName: \"kubernetes.io/projected/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-kube-api-access-4m7bg\") pod \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\" (UID: \"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7\") " Oct 30 01:00:18.770347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.770320 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdzc5\" (UniqueName: \"kubernetes.io/projected/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7-kube-api-access-wdzc5\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.776816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.776792 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-maas-api-service-ca" pod "47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" (UID: "47adc5ad-fcb5-4f50-864a-e0cd51b95bf7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:18.795830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.795788 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-kube-api-access-4m7bg" pod "47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" (UID: "47adc5ad-fcb5-4f50-864a-e0cd51b95bf7"). InnerVolumeSpecName "kube-api-access-4m7bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.856800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.856774 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" Oct 30 01:00:18.857688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.856806 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45" event={"ID":"1397a8c5-495a-4b7e-a747-e9220a1e0869","Type":"ContainerDied","Data":"652c0054572cf9545fff3e4cfae04ceb076e2c8efc4943b40784bb5f1a4a4dac"} Oct 30 01:00:18.857688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.857564 4905 scope.go:122] "RemoveContainer" containerID="64cc9d0b0c64ea07351b70a1be7702d5e769e6f33532abe641d62c5f65c9cddc" Oct 30 01:00:18.863574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.860593 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" event={"ID":"7e968a20-13f7-4bd2-8a94-ffa4d22dccb8","Type":"ContainerDied","Data":"57902452152d6e97880f8c6ae0ca463d89ec64a372cb3f83f4abe2e44913b1df"} Oct 30 01:00:18.863574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.860715 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv" Oct 30 01:00:18.865209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.865188 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" event={"ID":"455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7","Type":"ContainerDied","Data":"62c16d67325117168f7ff538ebf9e82a00c69d8480af2b6bf7660d7e149ba838"} Oct 30 01:00:18.865329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.865267 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr" Oct 30 01:00:18.869879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.869861 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" event={"ID":"47adc5ad-fcb5-4f50-864a-e0cd51b95bf7","Type":"ContainerDied","Data":"d95287583de8231f07caa9fbeebbca6f27e2745633a50640a96d621372bf0b7b"} Oct 30 01:00:18.870544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.869951 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v" Oct 30 01:00:18.872196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.870836 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1397a8c5-495a-4b7e-a747-e9220a1e0869-kube-api-access-qcl5m\" (UniqueName: \"kubernetes.io/projected/1397a8c5-495a-4b7e-a747-e9220a1e0869-kube-api-access-qcl5m\") pod \"1397a8c5-495a-4b7e-a747-e9220a1e0869\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " Oct 30 01:00:18.872196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.871195 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1397a8c5-495a-4b7e-a747-e9220a1e0869-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1397a8c5-495a-4b7e-a747-e9220a1e0869-maas-api-service-ca\") pod \"1397a8c5-495a-4b7e-a747-e9220a1e0869\" (UID: \"1397a8c5-495a-4b7e-a747-e9220a1e0869\") " Oct 30 01:00:18.872196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.871661 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1397a8c5-495a-4b7e-a747-e9220a1e0869-maas-api-service-ca" pod "1397a8c5-495a-4b7e-a747-e9220a1e0869" (UID: "1397a8c5-495a-4b7e-a747-e9220a1e0869"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:18.872196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.871902 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1397a8c5-495a-4b7e-a747-e9220a1e0869-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.872196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.871924 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.872196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.871939 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m7bg\" (UniqueName: \"kubernetes.io/projected/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7-kube-api-access-4m7bg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.876182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.875780 4905 scope.go:122] "RemoveContainer" containerID="1c7e8b726462272424a66abdd7280e313bed59436a79a419a741c61d5214b656" Oct 30 01:00:18.879182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.879157 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1397a8c5-495a-4b7e-a747-e9220a1e0869-kube-api-access-qcl5m" pod "1397a8c5-495a-4b7e-a747-e9220a1e0869" (UID: "1397a8c5-495a-4b7e-a747-e9220a1e0869"). InnerVolumeSpecName "kube-api-access-qcl5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.899043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.899012 4905 scope.go:122] "RemoveContainer" containerID="716d88d93a6bcb6d01018d6b8d8d517ee6ded7215e3d060f92f49163ddb46d0e" Oct 30 01:00:18.911512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.911431 4905 scope.go:122] "RemoveContainer" containerID="60b9156f053571f1599da079d5409629499bbd56aea2f4b0ab30915cbf8dd949" Oct 30 01:00:18.917064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.916853 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:18.978071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.976352 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcl5m\" (UniqueName: \"kubernetes.io/projected/1397a8c5-495a-4b7e-a747-e9220a1e0869-kube-api-access-qcl5m\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.985617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.985583 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v"] Oct 30 01:00:18.991041 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.991018 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-mdd8v"] Oct 30 01:00:18.998458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:18.998435 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:19.039433 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:19.039390 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5362aa_cf21_44ea_8e6d_cd3e540831e9.slice/crio-4a1c6e71c4cb07d903513ddd45dbc3c69bc0d95d0157715c3b8878fb287f2587 WatchSource:0}: Error finding container 4a1c6e71c4cb07d903513ddd45dbc3c69bc0d95d0157715c3b8878fb287f2587: Status 404 returned error can't find the container with id 4a1c6e71c4cb07d903513ddd45dbc3c69bc0d95d0157715c3b8878fb287f2587 Oct 30 01:00:19.042012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.041981 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr"] Oct 30 01:00:19.045115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.044594 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.045115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.044676 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.045115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.044717 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.052639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.052584 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888700-b5bbr"] Oct 30 01:00:19.072376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.072325 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc"] Oct 30 01:00:19.101843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.101807 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7"] Oct 30 01:00:19.184421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.184367 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.184592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.184473 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.184592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.184571 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.274129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.273564 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:19.289937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.288056 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" path="/var/lib/kubelet/pods/455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7/volumes" Oct 30 01:00:19.289937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.288573 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" path="/var/lib/kubelet/pods/47adc5ad-fcb5-4f50-864a-e0cd51b95bf7/volumes" Oct 30 01:00:19.289937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.289236 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv"] Oct 30 01:00:19.289937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.289270 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888700-rgqzv"] Oct 30 01:00:19.295556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.295539 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:19.347607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.347574 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ad6109-5703-409d-8441-798c75dae1f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad6109-5703-409d-8441-798c75dae1f8-maas-api-service-ca\") pod \"c1ad6109-5703-409d-8441-798c75dae1f8\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " Oct 30 01:00:19.347746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.347636 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ad6109-5703-409d-8441-798c75dae1f8-kube-api-access-6lwnr\" (UniqueName: \"kubernetes.io/projected/c1ad6109-5703-409d-8441-798c75dae1f8-kube-api-access-6lwnr\") pod \"c1ad6109-5703-409d-8441-798c75dae1f8\" (UID: \"c1ad6109-5703-409d-8441-798c75dae1f8\") " Oct 30 01:00:19.347856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.347776 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb2b8df-479a-417d-8105-035cb8a422a5-kube-api-access-5722f\" (UniqueName: \"kubernetes.io/projected/2eb2b8df-479a-417d-8105-035cb8a422a5-kube-api-access-5722f\") pod \"2eb2b8df-479a-417d-8105-035cb8a422a5\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " Oct 30 01:00:19.348007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.347991 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb2b8df-479a-417d-8105-035cb8a422a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb2b8df-479a-417d-8105-035cb8a422a5-maas-api-service-ca\") pod \"2eb2b8df-479a-417d-8105-035cb8a422a5\" (UID: \"2eb2b8df-479a-417d-8105-035cb8a422a5\") " Oct 30 01:00:19.349140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.349112 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ad6109-5703-409d-8441-798c75dae1f8-maas-api-service-ca" pod "c1ad6109-5703-409d-8441-798c75dae1f8" (UID: "c1ad6109-5703-409d-8441-798c75dae1f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.349653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.349632 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb2b8df-479a-417d-8105-035cb8a422a5-maas-api-service-ca" pod "2eb2b8df-479a-417d-8105-035cb8a422a5" (UID: "2eb2b8df-479a-417d-8105-035cb8a422a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.371679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.362838 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ad6109-5703-409d-8441-798c75dae1f8-kube-api-access-6lwnr" pod "c1ad6109-5703-409d-8441-798c75dae1f8" (UID: "c1ad6109-5703-409d-8441-798c75dae1f8"). InnerVolumeSpecName "kube-api-access-6lwnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.371679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.362917 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb2b8df-479a-417d-8105-035cb8a422a5-kube-api-access-5722f" pod "2eb2b8df-479a-417d-8105-035cb8a422a5" (UID: "2eb2b8df-479a-417d-8105-035cb8a422a5"). InnerVolumeSpecName "kube-api-access-5722f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.481151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.463470 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad6109-5703-409d-8441-798c75dae1f8-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.481151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.463539 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lwnr\" (UniqueName: \"kubernetes.io/projected/c1ad6109-5703-409d-8441-798c75dae1f8-kube-api-access-6lwnr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.481151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.463555 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5722f\" (UniqueName: \"kubernetes.io/projected/2eb2b8df-479a-417d-8105-035cb8a422a5-kube-api-access-5722f\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.481151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.463569 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb2b8df-479a-417d-8105-035cb8a422a5-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.509863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.493898 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45"] Oct 30 01:00:19.509863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.504710 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888700-nwg45"] Oct 30 01:00:19.714982 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.714268 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd"] Oct 30 01:00:19.750420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.746730 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67"] Oct 30 01:00:19.754245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.754204 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.779681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.779639 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.779947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.779920 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:19.988229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.986737 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" event={"ID":"dd5362aa-cf21-44ea-8e6d-cd3e540831e9","Type":"ContainerStarted","Data":"7c665c9a6c9aa43356537bb3fa26981c7b4f8a40e54a88b62ba91ed65345a934"} Oct 30 01:00:19.988229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.986798 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" event={"ID":"dd5362aa-cf21-44ea-8e6d-cd3e540831e9","Type":"ContainerStarted","Data":"4a1c6e71c4cb07d903513ddd45dbc3c69bc0d95d0157715c3b8878fb287f2587"} Oct 30 01:00:19.990131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.990103 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" event={"ID":"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47","Type":"ContainerStarted","Data":"ab276e351fc7b178d849ea6af4768dc199996b0c53636accda2e2cb59f44ac74"} Oct 30 01:00:19.997686 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.997658 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" event={"ID":"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e","Type":"ContainerStarted","Data":"f8cb959539addb1ead4650df739a0bbc182763d7de40224a4f7a07bf1b82c8d8"} Oct 30 01:00:19.999312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.999294 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" event={"ID":"2eb2b8df-479a-417d-8105-035cb8a422a5","Type":"ContainerDied","Data":"01b758fee28b48f851ecf429af30e1c466002d4d57f4b442cad08c2cf99ceed8"} Oct 30 01:00:19.999446 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.999437 4905 scope.go:122] "RemoveContainer" containerID="96af3d0f8362b85ea7a5bf993cfa4cbdd1d7da91e02f63abb140890ce55f4b54" Oct 30 01:00:19.999618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:19.999604 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq" Oct 30 01:00:20.012018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.008373 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" event={"ID":"a5bb9abe-dc1e-4805-9547-63c4606f6570","Type":"ContainerStarted","Data":"ea4010036801ff75c9b2040b20aebfdb992ad0e74f5428e965c429389741711d"} Oct 30 01:00:20.012018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.008473 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" event={"ID":"a5bb9abe-dc1e-4805-9547-63c4606f6570","Type":"ContainerStarted","Data":"f4c36b398d0b0a44a5db45e00c191646f966267905ff71f329c6e652f8b533d2"} Oct 30 01:00:20.053547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.027035 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" event={"ID":"c1ad6109-5703-409d-8441-798c75dae1f8","Type":"ContainerDied","Data":"0aad724bde5682ab580ba7ccbd20dbe3df0681aa95bf231a7c43bb406788ab93"} Oct 30 01:00:20.054372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.054328 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r" Oct 30 01:00:20.141545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.131379 4905 scope.go:122] "RemoveContainer" containerID="0f19d7da30a829db549576ffa1f107a7d2653a281074e78e37f86e7c6a8d3a58" Oct 30 01:00:20.278630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.261147 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podStartSLOduration=3.261068836 podStartE2EDuration="3.261068836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:20.088007651 +0000 UTC m=+1305.031180845" watchObservedRunningTime="2026-10-30 01:00:20.261068836 +0000 UTC m=+1305.204242028" Oct 30 01:00:20.278630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.278460 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:20.335100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.335064 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:20.439792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.434162 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:20.575961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.575895 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podStartSLOduration=3.575872544 podStartE2EDuration="3.575872544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:20.477446374 +0000 UTC m=+1305.420619567" watchObservedRunningTime="2026-10-30 01:00:20.575872544 +0000 UTC m=+1305.519045736" Oct 30 01:00:20.633622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.633574 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r"] Oct 30 01:00:20.694940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.694906 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888700-gls8r"] Oct 30 01:00:20.696666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.696642 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:20.700542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.696915 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:20.700754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.700727 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:20.756736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.756663 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq"] Oct 30 01:00:20.778037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:20.777982 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888700-dfggq"] Oct 30 01:00:21.144646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.144563 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" event={"ID":"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e","Type":"ContainerStarted","Data":"49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4"} Oct 30 01:00:21.145731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.145705 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:21.228195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.217446 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:21.265637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.239011 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" podStartSLOduration=3.23899191 podStartE2EDuration="3.23899191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:21.216756156 +0000 UTC m=+1306.159929348" watchObservedRunningTime="2026-10-30 01:00:21.23899191 +0000 UTC m=+1306.182165102" Oct 30 01:00:21.269055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.269015 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:21.273576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.269566 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:21.416600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.402991 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podStartSLOduration=4.40297119 podStartE2EDuration="4.40297119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:21.396093907 +0000 UTC m=+1306.339267103" watchObservedRunningTime="2026-10-30 01:00:21.40297119 +0000 UTC m=+1306.346144382" Oct 30 01:00:21.437707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.437663 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:21.439869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.439822 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:21.475221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.475186 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" path="/var/lib/kubelet/pods/1397a8c5-495a-4b7e-a747-e9220a1e0869/volumes" Oct 30 01:00:21.476396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.476374 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" path="/var/lib/kubelet/pods/2eb2b8df-479a-417d-8105-035cb8a422a5/volumes" Oct 30 01:00:21.477468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.477451 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" path="/var/lib/kubelet/pods/7e968a20-13f7-4bd2-8a94-ffa4d22dccb8/volumes" Oct 30 01:00:21.478827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.478810 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" path="/var/lib/kubelet/pods/c1ad6109-5703-409d-8441-798c75dae1f8/volumes" Oct 30 01:00:21.479620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.479586 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" event={"ID":"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47","Type":"ContainerStarted","Data":"52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b"} Oct 30 01:00:21.700213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.700117 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:21.700503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.700460 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:21.925562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.921457 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:21.925562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.921625 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:21.925562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.921671 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:21.941671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.941619 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:21.943386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.943347 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:21.970946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:21.970854 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:00:22.117541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.109724 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:00:22.208582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.208536 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:22.234718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.234672 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:00:22.235560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.235511 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:00:22.270903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.270867 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:22.271155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.271121 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:22.668745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.469448 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:22.669462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.669436 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:00:22.675018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.480144 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:00:22.676377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.526430 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:22.708458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.708422 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:22.722679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.708690 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:22.905430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.894749 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:00:22.957425 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.952432 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:22.957425 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.953017 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:00:22.971630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.971534 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:22.972545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.971899 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:00:22.972657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.972612 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:00:22.995401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:22.995371 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:23.011271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.010811 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:23.081749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.081711 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:00:23.396544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.383886 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:23.437168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.437127 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:00:23.539411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.539370 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:00:23.561553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.557015 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:23.561553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.557075 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:23.684621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.681176 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:00:23.684621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.681989 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:23.913548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.812652 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:23.913548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:23.812733 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:24.026557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.008617 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 01:00:24.026557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.008912 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:00:24.026557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.010429 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:24.026557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.010482 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:24.026557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.010917 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:00:24.524972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.509409 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:24.524972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.523952 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:00:24.551569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.531095 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:00:24.606558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.557853 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:00:24.661137 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.454614 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:00:24.680562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.664581 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:00:24.683282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.683241 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:24.699689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.699631 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:00:24.737574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.735489 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:24.737574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.735567 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.785405 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.785464 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:24.791376 4905 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.42s" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791455 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791567 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791616 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791653 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791692 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791735 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791767 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:00:24.793129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:24.791797 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:00:25.023072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.023032 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:25.093444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.093354 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:25.093733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.093703 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:25.139683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.139643 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz"] Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186871 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186902 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186924 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186946 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186968 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186975 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186991 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.193545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.186998 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.293942 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.293914 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:25.294182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.294165 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:25.294305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.294294 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.294896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.294879 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.311989 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.311964 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.312172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312157 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.312309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312297 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312391 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312408 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312416 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312432 4905 cpu_manager.go:414] "containerMap: removing container" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312439 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312462 4905 cpu_manager.go:414] "containerMap: removing container" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312469 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312494 4905 cpu_manager.go:414] "containerMap: removing container" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312501 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312510 4905 cpu_manager.go:414] "containerMap: removing container" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312570 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.312628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.312601 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:25.313619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.313547 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:25.313619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.313573 4905 cpu_manager.go:414] "containerMap: removing container" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.313619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.313580 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.314617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.314577 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.314773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.314734 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:25.314978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.314755 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.314978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.314926 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:25.314978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.314940 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:25.314978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.314949 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.315306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.314959 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:25.315306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.315279 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:25.315543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.315444 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:25.315543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.315478 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="47adc5ad-fcb5-4f50-864a-e0cd51b95bf7" containerName="cleanup" Oct 30 01:00:25.315543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.315492 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:25.315543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.315500 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.315861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.315513 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:25.315861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.315825 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:25.318814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.318791 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:25.318958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.318939 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:25.319285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.262904 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:25.345773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.326167 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.390562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.384830 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:00:25.409648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.409621 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:00:25.509945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.478553 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rxx\" (UniqueName: \"kubernetes.io/projected/3d8fadf7-9f08-4979-9aee-11185d539e50-kube-api-access-d9rxx\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.509945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.478871 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.509945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.486438 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:00:25.509945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.487640 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:00:25.513345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.513314 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.548543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.537186 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:25.736601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.544846 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:00:25.736601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.545463 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:25.736601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.735835 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:00:25.736601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.614815 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:25.737408 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.672882 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.737613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.737594 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.737701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.737681 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rxx\" (UniqueName: \"kubernetes.io/projected/3d8fadf7-9f08-4979-9aee-11185d539e50-kube-api-access-d9rxx\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.737786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.696354 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:25.737854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.737810 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:25.738128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.673737 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.738216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.673790 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:25.844042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.702603 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:00:25.847778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.673833 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:00:25.848463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.825164 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:25.848604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:25.829577 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz"] Oct 30 01:00:26.041597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.029206 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:00:26.043477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.040638 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:26.198760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.198726 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:26.199032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.199002 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:26.232271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.210170 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rxx\" (UniqueName: \"kubernetes.io/projected/3d8fadf7-9f08-4979-9aee-11185d539e50-kube-api-access-d9rxx\") pod \"maas-api-revoke-sub-99549ef3-srgdz\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:26.268135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.267499 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:26.268135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.267571 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:26.331616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.323739 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:26.395860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.392979 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:00:26.469541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.467290 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:26.744543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.735601 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:00:26.761086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.760967 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 01:00:26.786633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.786548 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:00:26.795722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.795639 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:00:26.867624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.863715 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:26.943595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.918813 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:00:26.970546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.875622 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:00:27.542612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.987718 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:00:27.564904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.988898 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:27.603902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.603805 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:27.641068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.060492 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:27.641661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.370129 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:27.681543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.059641 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:00:27.682404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:26.989118 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:27.708413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.708373 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:27.708754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.300508 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:27.724544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.708877 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:27.822033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.821990 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:27.895589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.867846 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:00:27.895589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.887217 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:27.895589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.887271 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:27.911634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.901926 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:27.951700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.948592 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:27.986647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.967955 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:27.986647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:27.968018 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:28.034662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.019292 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:00:28.034662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.019404 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:00:28.034662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.019467 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:00:28.301542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.289928 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:28.301542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.289993 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:28.620352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.620271 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:00:28.640061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.640033 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:00:28.705546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.682829 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:28.727541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.718884 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:28.727541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.718940 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:28.824159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.824099 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:28.870696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.870601 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:28.898662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.887282 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx"] Oct 30 01:00:28.900712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.887391 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:28.902477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.902456 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:28.908725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.908704 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:28.908942 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.908907 4905 cpu_manager.go:414] "containerMap: removing container" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:28.909071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.909054 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455d2a35-b7c5-4b1f-8b1c-3048f14f6ec7" containerName="cleanup" Oct 30 01:00:28.909259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.909246 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:28.909379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.909366 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:28.926702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.922313 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:28.929166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.904935 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:28.977331 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.927198 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ad6109-5703-409d-8441-798c75dae1f8" containerName="cleanup" Oct 30 01:00:28.999152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.999120 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb2b8df-479a-417d-8105-035cb8a422a5" containerName="cleanup" Oct 30 01:00:28.999382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.999366 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1397a8c5-495a-4b7e-a747-e9220a1e0869" containerName="cleanup" Oct 30 01:00:28.999507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.999494 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e968a20-13f7-4bd2-8a94-ffa4d22dccb8" containerName="cleanup" Oct 30 01:00:29.045546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:28.997183 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:29.045546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.029630 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:29.045546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.030046 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:29.178597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.155406 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx"] Oct 30 01:00:29.178597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.155567 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.178597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.175493 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:29.402399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.402367 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:29.402666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.402642 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:29.410152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.410130 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:00:29.439600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.436577 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t592g\" (UniqueName: \"kubernetes.io/projected/19bf8159-6bb7-4791-aa29-7e59dd281033-kube-api-access-t592g\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.439600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.436693 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.439600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.436931 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.504369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.503322 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:00:29.575928 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.575892 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t592g\" (UniqueName: \"kubernetes.io/projected/19bf8159-6bb7-4791-aa29-7e59dd281033-kube-api-access-t592g\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.576209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.576189 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.576564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.576537 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.605959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.605928 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.612501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.611126 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t592g\" (UniqueName: \"kubernetes.io/projected/19bf8159-6bb7-4791-aa29-7e59dd281033-kube-api-access-t592g\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:29.615553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.615184 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:00:29.615886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.615867 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:00:29.619474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.619450 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:00:29.722760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.722674 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:29.724005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.723973 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:29.974009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.973467 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:29.977054 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:29.973543 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:30.054589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.054558 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eb2911f-gvjpx\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:30.141389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.097503 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz"] Oct 30 01:00:30.268131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.267494 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:30.268131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.267658 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:30.332356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.327962 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:30.347141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.347098 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" event={"ID":"3d8fadf7-9f08-4979-9aee-11185d539e50","Type":"ContainerStarted","Data":"aa8407665e86b2f39bc7bf4fd2480deb466a4c60acabb510d8c098305a650b8c"} Oct 30 01:00:30.693661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.693621 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:30.693860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.693689 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:30.773592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.773105 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx"] Oct 30 01:00:30.782543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.780895 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:30.782543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.780952 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:30.918853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.918810 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:30.919017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:30.918867 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:31.268671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.268095 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zz5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 01:00:31.268671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.268160 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 01:00:31.400434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.399262 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" event={"ID":"19bf8159-6bb7-4791-aa29-7e59dd281033","Type":"ContainerStarted","Data":"1870f08dad55041a8436964c39fac99ff39411a50f71e1a259bb1dfb7798c1b9"} Oct 30 01:00:31.404166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.404129 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" event={"ID":"3d8fadf7-9f08-4979-9aee-11185d539e50","Type":"ContainerStarted","Data":"c09ab90338466742383a14735c11421b8c5cdc16c569f1cc6a9da61c05603221"} Oct 30 01:00:31.452325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.452248 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" podStartSLOduration=7.452228517 podStartE2EDuration="7.452228517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:31.44536584 +0000 UTC m=+1316.388539033" watchObservedRunningTime="2026-10-30 01:00:31.452228517 +0000 UTC m=+1316.395401709" Oct 30 01:00:31.697599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.694890 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-96bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:00:31.697599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.694948 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 01:00:31.918428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.918382 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:00:31.918633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:31.918478 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:00:32.423950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:32.423806 4905 generic.go:356] "Generic (PLEG): container finished" podID="19bf8159-6bb7-4791-aa29-7e59dd281033" containerID="180449f331f3a878849337c9aa55069dc6e1021f916c1c10db7c691261b48b47" exitCode=0 Oct 30 01:00:32.424452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:32.424046 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" event={"ID":"19bf8159-6bb7-4791-aa29-7e59dd281033","Type":"ContainerDied","Data":"180449f331f3a878849337c9aa55069dc6e1021f916c1c10db7c691261b48b47"} Oct 30 01:00:32.450543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:32.450413 4905 generic.go:356] "Generic (PLEG): container finished" podID="3d8fadf7-9f08-4979-9aee-11185d539e50" containerID="c09ab90338466742383a14735c11421b8c5cdc16c569f1cc6a9da61c05603221" exitCode=0 Oct 30 01:00:32.450727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:32.450668 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" event={"ID":"3d8fadf7-9f08-4979-9aee-11185d539e50","Type":"ContainerDied","Data":"c09ab90338466742383a14735c11421b8c5cdc16c569f1cc6a9da61c05603221"} Oct 30 01:00:32.885543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:32.885171 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:32.889221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:32.886619 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:32.889221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:32.887794 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:00:33.455737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:33.452750 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:33.455737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:33.455454 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:33.460134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:33.459986 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:00:33.635339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:33.633457 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:33.635339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:33.635128 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:33.638646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:33.638595 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:00:33.900814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:33.899918 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:34.036097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.036062 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:34.043315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.043284 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bf8159-6bb7-4791-aa29-7e59dd281033-kube-api-access-t592g\" (UniqueName: \"kubernetes.io/projected/19bf8159-6bb7-4791-aa29-7e59dd281033-kube-api-access-t592g\") pod \"19bf8159-6bb7-4791-aa29-7e59dd281033\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " Oct 30 01:00:34.043449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.043398 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-openshift-ca\") pod \"19bf8159-6bb7-4791-aa29-7e59dd281033\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " Oct 30 01:00:34.043632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.043611 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-xks-ca\") pod \"19bf8159-6bb7-4791-aa29-7e59dd281033\" (UID: \"19bf8159-6bb7-4791-aa29-7e59dd281033\") " Oct 30 01:00:34.043965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.043844 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-openshift-ca" pod "19bf8159-6bb7-4791-aa29-7e59dd281033" (UID: "19bf8159-6bb7-4791-aa29-7e59dd281033"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:34.044478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.044454 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:34.046703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.046666 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-xks-ca" pod "19bf8159-6bb7-4791-aa29-7e59dd281033" (UID: "19bf8159-6bb7-4791-aa29-7e59dd281033"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:00:34.046793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.046769 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bf8159-6bb7-4791-aa29-7e59dd281033-kube-api-access-t592g" pod "19bf8159-6bb7-4791-aa29-7e59dd281033" (UID: "19bf8159-6bb7-4791-aa29-7e59dd281033"). InnerVolumeSpecName "kube-api-access-t592g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:34.145352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.145261 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-openshift-ca\") pod \"3d8fadf7-9f08-4979-9aee-11185d539e50\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " Oct 30 01:00:34.145510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.145411 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8fadf7-9f08-4979-9aee-11185d539e50-kube-api-access-d9rxx\" (UniqueName: \"kubernetes.io/projected/3d8fadf7-9f08-4979-9aee-11185d539e50-kube-api-access-d9rxx\") pod \"3d8fadf7-9f08-4979-9aee-11185d539e50\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " Oct 30 01:00:34.145510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.145442 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-xks-ca\") pod \"3d8fadf7-9f08-4979-9aee-11185d539e50\" (UID: \"3d8fadf7-9f08-4979-9aee-11185d539e50\") " Oct 30 01:00:34.145734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.145704 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-openshift-ca" pod "3d8fadf7-9f08-4979-9aee-11185d539e50" (UID: "3d8fadf7-9f08-4979-9aee-11185d539e50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:34.146090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.146070 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:34.146167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.146092 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t592g\" (UniqueName: \"kubernetes.io/projected/19bf8159-6bb7-4791-aa29-7e59dd281033-kube-api-access-t592g\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:34.146167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.146101 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bf8159-6bb7-4791-aa29-7e59dd281033-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:34.148242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.148210 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8fadf7-9f08-4979-9aee-11185d539e50-kube-api-access-d9rxx" pod "3d8fadf7-9f08-4979-9aee-11185d539e50" (UID: "3d8fadf7-9f08-4979-9aee-11185d539e50"). InnerVolumeSpecName "kube-api-access-d9rxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:34.148242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.148233 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-xks-ca" pod "3d8fadf7-9f08-4979-9aee-11185d539e50" (UID: "3d8fadf7-9f08-4979-9aee-11185d539e50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:00:34.247777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.247744 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9rxx\" (UniqueName: \"kubernetes.io/projected/3d8fadf7-9f08-4979-9aee-11185d539e50-kube-api-access-d9rxx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:34.248037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.248012 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8fadf7-9f08-4979-9aee-11185d539e50-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:34.485191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.485167 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" Oct 30 01:00:34.485715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.485205 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz" event={"ID":"3d8fadf7-9f08-4979-9aee-11185d539e50","Type":"ContainerDied","Data":"aa8407665e86b2f39bc7bf4fd2480deb466a4c60acabb510d8c098305a650b8c"} Oct 30 01:00:34.485715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.485261 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8407665e86b2f39bc7bf4fd2480deb466a4c60acabb510d8c098305a650b8c" Oct 30 01:00:34.487109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.487090 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" Oct 30 01:00:34.487217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.487157 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx" event={"ID":"19bf8159-6bb7-4791-aa29-7e59dd281033","Type":"ContainerDied","Data":"1870f08dad55041a8436964c39fac99ff39411a50f71e1a259bb1dfb7798c1b9"} Oct 30 01:00:34.487217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.487206 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1870f08dad55041a8436964c39fac99ff39411a50f71e1a259bb1dfb7798c1b9" Oct 30 01:00:34.788994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:34.788904 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:00:36.591765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.591728 4905 generic.go:356] "Generic (PLEG): container finished" podID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerID="9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7" exitCode=0 Oct 30 01:00:36.592242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.591803 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" event={"ID":"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290","Type":"ContainerDied","Data":"9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7"} Oct 30 01:00:36.592242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.592165 4905 scope.go:122] "RemoveContainer" containerID="9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7" Oct 30 01:00:36.593638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.593610 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:36.593712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.593680 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:36.593749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.593714 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:36.797902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.797868 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6"] Oct 30 01:00:36.798695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.798673 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3d8fadf7-9f08-4979-9aee-11185d539e50" containerName="revoke-keys" Oct 30 01:00:36.798695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.798694 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8fadf7-9f08-4979-9aee-11185d539e50" containerName="revoke-keys" Oct 30 01:00:36.798867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.798713 4905 cpu_manager.go:414] "containerMap: removing container" podUID="19bf8159-6bb7-4791-aa29-7e59dd281033" containerName="revoke-keys" Oct 30 01:00:36.798867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.798721 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bf8159-6bb7-4791-aa29-7e59dd281033" containerName="revoke-keys" Oct 30 01:00:36.798971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.798941 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8fadf7-9f08-4979-9aee-11185d539e50" containerName="revoke-keys" Oct 30 01:00:36.798971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.798958 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bf8159-6bb7-4791-aa29-7e59dd281033" containerName="revoke-keys" Oct 30 01:00:36.803019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.802997 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:36.819964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.819928 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:00:36.820082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.819984 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:00:36.820082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.820007 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:00:36.820082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.820038 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:00:36.820478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:36.820458 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6"] Oct 30 01:00:37.112961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.101460 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnw8l\" (UniqueName: \"kubernetes.io/projected/7131451d-9f5d-48b9-bf05-11d18be41535-kube-api-access-hnw8l\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.221559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.219831 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.221559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.220039 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.270681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.245885 4905 scope.go:122] "RemoveContainer" containerID="00bd05b48712372056a3c4679dce03905daf711ae3fa2dfc8fd6c9f1f5e68ea3" Oct 30 01:00:37.350828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.350736 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnw8l\" (UniqueName: \"kubernetes.io/projected/7131451d-9f5d-48b9-bf05-11d18be41535-kube-api-access-hnw8l\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.350828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.350802 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.351094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.350937 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.373623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.365709 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.487782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.487735 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnw8l\" (UniqueName: \"kubernetes.io/projected/7131451d-9f5d-48b9-bf05-11d18be41535-kube-api-access-hnw8l\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.492552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.489957 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d49641-br5j6\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.612713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.612687 4905 scope.go:122] "RemoveContainer" containerID="bdd80bcca47dd5928d4a97cb7c4d1362722fbba521bf51f4934ad6b50097b2dc" Oct 30 01:00:37.664035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.664009 4905 scope.go:122] "RemoveContainer" containerID="2305d6fe776b97adeefd4b4daa7a156186448454a07dbb47330dfa52e22b0332" Oct 30 01:00:37.668632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.668599 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" event={"ID":"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290","Type":"ContainerStarted","Data":"cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849"} Oct 30 01:00:37.686985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.686967 4905 scope.go:122] "RemoveContainer" containerID="c8e087981e44614c3987feff4512fe4458a4a9f108bc1391d76038a455a58630" Oct 30 01:00:37.707492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.707325 4905 scope.go:122] "RemoveContainer" containerID="ac1b411b622a21fae6623f7dd241348b523b4092bde1d2c9492a44170bb30df7" Oct 30 01:00:37.719913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.719884 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 01:00:37.720205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.720178 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:37.721242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.721208 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:00:37.721343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.721278 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 01:00:37.726608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.726553 4905 scope.go:122] "RemoveContainer" containerID="1df0d2863f1e22d4583edacfe9270ff71bc7bd37bc8d2041b8fd357a3e974729" Oct 30 01:00:37.759858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.749310 4905 scope.go:122] "RemoveContainer" containerID="6217323f3155c5e8361331e091a3ca9b6e38299b009fc7d753853aac96a23e95" Oct 30 01:00:37.893871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:37.887144 4905 scope.go:122] "RemoveContainer" containerID="e6bdfe2cebded9c46ed12836d4b271311e3832908524a8867f69980fdef508f1" Oct 30 01:00:38.023813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:38.001022 4905 scope.go:122] "RemoveContainer" containerID="e0abab8fe817b126192ce56a5dc1f59f9ae2047e03230e0cca454af12c53fbf5" Oct 30 01:00:38.069693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:38.069238 4905 scope.go:122] "RemoveContainer" containerID="a2fdf01e6e94b9a2082891b8f2bffae08715a23e24ba1ddcd5f1261b53ecfd47" Oct 30 01:00:38.092063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:38.092002 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76"] Oct 30 01:00:38.103036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:38.103014 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7500e0-kxc76"] Oct 30 01:00:38.636597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:38.636570 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6"] Oct 30 01:00:38.782598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:38.781292 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:00:38.782598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:38.781359 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 01:00:39.161955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:39.158342 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" event={"ID":"7131451d-9f5d-48b9-bf05-11d18be41535","Type":"ContainerStarted","Data":"31d852f2a4447dcd9f09700678b30d88739602342f73e9877eccc2c38cd12dbc"} Oct 30 01:00:39.326285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:39.326204 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b45434-400b-41cd-8843-d10eb66b5154" path="/var/lib/kubelet/pods/91b45434-400b-41cd-8843-d10eb66b5154/volumes" Oct 30 01:00:39.720857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:39.720822 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:00:39.721423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:39.721382 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 01:00:39.866746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:39.866694 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.017645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.010609 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.045600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.044601 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.350692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.324501 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" event={"ID":"7131451d-9f5d-48b9-bf05-11d18be41535","Type":"ContainerStarted","Data":"90b49f0040a618ce644d1c806df919b10ab109849ed1e3400578404f0a2312e3"} Oct 30 01:00:40.414799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.400144 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" podStartSLOduration=4.400124565 podStartE2EDuration="4.400124565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:40.384961428 +0000 UTC m=+1325.328134633" watchObservedRunningTime="2026-10-30 01:00:40.400124565 +0000 UTC m=+1325.343297758" Oct 30 01:00:40.669703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.669593 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.725031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.724997 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:00:40.740967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.740869 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 01:00:40.800554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:40.792595 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:00:41.023553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.022599 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.023553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.023173 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:00:41.065799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.065756 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.066820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.066785 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:00:41.170005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.169969 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5"] Oct 30 01:00:41.512217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.512172 4905 generic.go:356] "Generic (PLEG): container finished" podID="7131451d-9f5d-48b9-bf05-11d18be41535" containerID="90b49f0040a618ce644d1c806df919b10ab109849ed1e3400578404f0a2312e3" exitCode=0 Oct 30 01:00:41.659947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.659892 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.707756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.707660 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5696c81b-s8ff5"] Oct 30 01:00:41.711898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.711866 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" event={"ID":"7131451d-9f5d-48b9-bf05-11d18be41535","Type":"ContainerDied","Data":"90b49f0040a618ce644d1c806df919b10ab109849ed1e3400578404f0a2312e3"} Oct 30 01:00:41.852601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.849667 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:00:41.875538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.676585 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:00:41.875538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.753625 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.875538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.757592 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.875538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.870673 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:00:41.875538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.691055 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 01:00:41.898895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.898828 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:00:41.956546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.904295 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:00:41.956546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:41.945293 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 01:00:42.115743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.115215 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:00:42.415206 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:00:42.415132 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:00:42.583751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.581575 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh"] Oct 30 01:00:42.590644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.589919 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.603949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.603763 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh"] Oct 30 01:00:42.607763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.607738 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-istio-token\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.608031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608011 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-data\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.608168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608153 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc9348d-15c2-410a-bcd7-3781c7888192-istiod-ca-cert\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.608299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608285 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-certs\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.608447 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608433 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-envoy\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.608567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608553 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-socket\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.608780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608764 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc9348d-15c2-410a-bcd7-3781c7888192-istio-podinfo\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.608899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608885 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s59g\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-kube-api-access-7s59g\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.609108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.609092 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-credential-socket\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.609563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.608495 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fcc0d3e0-openshift-default-dockercfg-xmx7c\"" Oct 30 01:00:42.711402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.711371 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc9348d-15c2-410a-bcd7-3781c7888192-istio-podinfo\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.711677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.711644 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s59g\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-kube-api-access-7s59g\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.711953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.711923 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-credential-socket\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.712129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.712099 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-istio-token\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.712315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.712268 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-credential-socket\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.712603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.712579 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-data\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.712701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.712639 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-data\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.713501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.712685 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc9348d-15c2-410a-bcd7-3781c7888192-istiod-ca-cert\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.713620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.713549 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-certs\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.713620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.713592 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-envoy\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.713620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.713611 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-socket\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.713843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.713826 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc9348d-15c2-410a-bcd7-3781c7888192-istio-podinfo\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.713896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.713862 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-socket\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.713952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.713934 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc9348d-15c2-410a-bcd7-3781c7888192-istiod-ca-cert\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.714020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.714005 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-certs\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.717721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.716112 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-envoy\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:42.719953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.719365 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:00:42.719953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:42.719410 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 01:00:43.009276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.009247 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s59g\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-kube-api-access-7s59g\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:43.010224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.010176 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-istio-token\") pod \"e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:43.094840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.094807 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:43.127148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.127113 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7"] Oct 30 01:00:43.217832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.214200 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:43.289373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.289339 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-openshift-ca\") pod \"7131451d-9f5d-48b9-bf05-11d18be41535\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " Oct 30 01:00:43.289687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.289666 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-xks-ca\") pod \"7131451d-9f5d-48b9-bf05-11d18be41535\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " Oct 30 01:00:43.290082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.290065 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7131451d-9f5d-48b9-bf05-11d18be41535-kube-api-access-hnw8l\" (UniqueName: \"kubernetes.io/projected/7131451d-9f5d-48b9-bf05-11d18be41535-kube-api-access-hnw8l\") pod \"7131451d-9f5d-48b9-bf05-11d18be41535\" (UID: \"7131451d-9f5d-48b9-bf05-11d18be41535\") " Oct 30 01:00:43.314245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.304918 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-openshift-ca" pod "7131451d-9f5d-48b9-bf05-11d18be41535" (UID: "7131451d-9f5d-48b9-bf05-11d18be41535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:43.478594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.463422 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:43.478594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.476504 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8655488f-e05c-4647-95dd-c20d4703aee4" path="/var/lib/kubelet/pods/8655488f-e05c-4647-95dd-c20d4703aee4/volumes" Oct 30 01:00:43.479938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.479915 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd33ef9b-4qpz7"] Oct 30 01:00:43.488976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.486942 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-xks-ca" pod "7131451d-9f5d-48b9-bf05-11d18be41535" (UID: "7131451d-9f5d-48b9-bf05-11d18be41535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:00:43.488976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.488919 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7131451d-9f5d-48b9-bf05-11d18be41535-kube-api-access-hnw8l" pod "7131451d-9f5d-48b9-bf05-11d18be41535" (UID: "7131451d-9f5d-48b9-bf05-11d18be41535"). InnerVolumeSpecName "kube-api-access-hnw8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:43.571562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.571083 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7131451d-9f5d-48b9-bf05-11d18be41535-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:43.571562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.571110 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnw8l\" (UniqueName: \"kubernetes.io/projected/7131451d-9f5d-48b9-bf05-11d18be41535-kube-api-access-hnw8l\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:00:43.579512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.579472 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" event={"ID":"7131451d-9f5d-48b9-bf05-11d18be41535","Type":"ContainerDied","Data":"31d852f2a4447dcd9f09700678b30d88739602342f73e9877eccc2c38cd12dbc"} Oct 30 01:00:43.579666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.579566 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d852f2a4447dcd9f09700678b30d88739602342f73e9877eccc2c38cd12dbc" Oct 30 01:00:43.579666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.579641 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6" Oct 30 01:00:43.718818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.718787 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:00:43.718984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.718845 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 01:00:43.771826 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:43.769411 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc9348d_15c2_410a_bcd7_3781c7888192.slice/crio-2192928904ea9d5c2a3f6564b22e36b775903dc790d05605f565a3f1a5341515 WatchSource:0}: Error finding container 2192928904ea9d5c2a3f6564b22e36b775903dc790d05605f565a3f1a5341515: Status 404 returned error can't find the container with id 2192928904ea9d5c2a3f6564b22e36b775903dc790d05605f565a3f1a5341515 Oct 30 01:00:43.771826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.771696 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh"] Oct 30 01:00:43.772817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.772783 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:43.772935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.772865 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:43.773010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:43.772910 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:00:44.591797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:44.591756 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" event={"ID":"edc9348d-15c2-410a-bcd7-3781c7888192","Type":"ContainerStarted","Data":"5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0"} Oct 30 01:00:44.592209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:44.591813 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" event={"ID":"edc9348d-15c2-410a-bcd7-3781c7888192","Type":"ContainerStarted","Data":"2192928904ea9d5c2a3f6564b22e36b775903dc790d05605f565a3f1a5341515"} Oct 30 01:00:44.651013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:44.650950 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podStartSLOduration=2.65093363 podStartE2EDuration="2.65093363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:44.650740535 +0000 UTC m=+1329.593913727" watchObservedRunningTime="2026-10-30 01:00:44.65093363 +0000 UTC m=+1329.594106824" Oct 30 01:00:44.715507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:44.715466 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 01:00:44.751511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:44.751485 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 01:00:45.193088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:45.193054 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7918536-7d0d-4835-a8f2-f2ef416c7b7b" path="/var/lib/kubelet/pods/b7918536-7d0d-4835-a8f2-f2ef416c7b7b/volumes" Oct 30 01:00:45.218118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:45.218087 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:45.223140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:45.223123 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:45.600027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:45.599983 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:45.600797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:45.600770 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:00:45.600927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:45.600828 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 01:00:49.681945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.681915 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4"] Oct 30 01:00:49.682674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.682653 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7131451d-9f5d-48b9-bf05-11d18be41535" containerName="revoke-keys" Oct 30 01:00:49.682674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.682672 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7131451d-9f5d-48b9-bf05-11d18be41535" containerName="revoke-keys" Oct 30 01:00:49.682838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.682803 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7131451d-9f5d-48b9-bf05-11d18be41535" containerName="revoke-keys" Oct 30 01:00:49.687507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.687488 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:49.707997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.707969 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4"] Oct 30 01:00:49.712729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.712705 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fcc0d3e0\"" Oct 30 01:00:49.739917 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.739879 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d"] Oct 30 01:00:49.744883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.744862 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:49.766305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.765805 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d"] Oct 30 01:00:49.797042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.797009 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mnq\" (UniqueName: \"kubernetes.io/projected/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-kube-api-access-v7mnq\") pod \"payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:49.797315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.797282 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:49.797449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.797331 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwdb\" (UniqueName: \"kubernetes.io/projected/e8af2efd-365f-4419-8f25-8e7276bedd83-kube-api-access-vhwdb\") pod \"payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:49.797449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.797399 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af2efd-365f-4419-8f25-8e7276bedd83-plugins-config-volume\") pod \"payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:49.803418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.803375 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fcc0d3e0-dockercfg-gq7xb\"" Oct 30 01:00:49.953596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.906462 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:49.953596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.906531 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwdb\" (UniqueName: \"kubernetes.io/projected/e8af2efd-365f-4419-8f25-8e7276bedd83-kube-api-access-vhwdb\") pod \"payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:49.953596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.906607 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af2efd-365f-4419-8f25-8e7276bedd83-plugins-config-volume\") pod \"payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:49.953596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.906729 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mnq\" (UniqueName: \"kubernetes.io/projected/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-kube-api-access-v7mnq\") pod \"payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:49.953596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.908393 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:49.957550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:49.954967 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af2efd-365f-4419-8f25-8e7276bedd83-plugins-config-volume\") pod \"payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:50.057678 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:50.021194 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mnq\" (UniqueName: \"kubernetes.io/projected/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-kube-api-access-v7mnq\") pod \"payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:50.124553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:50.108405 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwdb\" (UniqueName: \"kubernetes.io/projected/e8af2efd-365f-4419-8f25-8e7276bedd83-kube-api-access-vhwdb\") pod \"payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:50.125086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:50.124824 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:50.340637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:50.338184 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:50.977177 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:50.977134 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8af2efd_365f_4419_8f25_8e7276bedd83.slice/crio-2fe4eb26d5c824ded80d51f1603b4f82c6f68362ada54a276d6bcb9973fb1e7b WatchSource:0}: Error finding container 2fe4eb26d5c824ded80d51f1603b4f82c6f68362ada54a276d6bcb9973fb1e7b: Status 404 returned error can't find the container with id 2fe4eb26d5c824ded80d51f1603b4f82c6f68362ada54a276d6bcb9973fb1e7b Oct 30 01:00:50.981698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:50.981674 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4"] Oct 30 01:00:50.990167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:50.990147 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d"] Oct 30 01:00:50.992882 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:50.992858 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0974e7dc_5e06_4e10_bf33_cfc27a12ea96.slice/crio-086f06ab2f966a5a29466009500b78828ef01b7e9004f36dffe04c15eccef0df WatchSource:0}: Error finding container 086f06ab2f966a5a29466009500b78828ef01b7e9004f36dffe04c15eccef0df: Status 404 returned error can't find the container with id 086f06ab2f966a5a29466009500b78828ef01b7e9004f36dffe04c15eccef0df Oct 30 01:00:51.733920 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.733884 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" event={"ID":"0974e7dc-5e06-4e10-bf33-cfc27a12ea96","Type":"ContainerStarted","Data":"c7e65a8980b37b0286056e975421af3ec0aa4b0d7a1c2449a1bdb7c18a4c67db"} Oct 30 01:00:51.734129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.733956 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:00:51.734129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.733975 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" event={"ID":"0974e7dc-5e06-4e10-bf33-cfc27a12ea96","Type":"ContainerStarted","Data":"086f06ab2f966a5a29466009500b78828ef01b7e9004f36dffe04c15eccef0df"} Oct 30 01:00:51.735756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.735730 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" event={"ID":"e8af2efd-365f-4419-8f25-8e7276bedd83","Type":"ContainerStarted","Data":"932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39"} Oct 30 01:00:51.735886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.735780 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" event={"ID":"e8af2efd-365f-4419-8f25-8e7276bedd83","Type":"ContainerStarted","Data":"2fe4eb26d5c824ded80d51f1603b4f82c6f68362ada54a276d6bcb9973fb1e7b"} Oct 30 01:00:51.735886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.735810 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:00:51.776311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.776259 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" podStartSLOduration=2.776241749 podStartE2EDuration="2.776241749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:51.773717259 +0000 UTC m=+1336.716890452" watchObservedRunningTime="2026-10-30 01:00:51.776241749 +0000 UTC m=+1336.719414943" Oct 30 01:00:51.818190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:51.818149 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" podStartSLOduration=2.818137315 podStartE2EDuration="2.818137315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:51.813119041 +0000 UTC m=+1336.756292235" watchObservedRunningTime="2026-10-30 01:00:51.818137315 +0000 UTC m=+1336.761310507" Oct 30 01:00:52.054595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:52.053669 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt"] Oct 30 01:00:52.067405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:52.066327 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfacb692-gvgqt"] Oct 30 01:00:53.031900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:53.031864 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b"] Oct 30 01:00:53.045363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:53.045336 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1609a64-xcx5b"] Oct 30 01:00:53.192495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:53.192456 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f" path="/var/lib/kubelet/pods/26cbb50b-8b3e-4f5a-8c44-4a8640c66e4f/volumes" Oct 30 01:00:53.192973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:53.192956 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8754e4f1-dcf3-44d0-a04d-09b44b421a9f" path="/var/lib/kubelet/pods/8754e4f1-dcf3-44d0-a04d-09b44b421a9f/volumes" Oct 30 01:00:54.223463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.223428 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx"] Oct 30 01:00:54.229699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.229668 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:54.252452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.252428 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx"] Oct 30 01:00:54.255265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.255238 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7452d0-dockercfg-wdlrx\"" Oct 30 01:00:54.281085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.281052 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7452d0\"" Oct 30 01:00:54.430551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.378047 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4574\" (UniqueName: \"kubernetes.io/projected/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-kube-api-access-d4574\") pod \"payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:54.430551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.378329 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:54.481603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.480764 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:54.481603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.481065 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4574\" (UniqueName: \"kubernetes.io/projected/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-kube-api-access-d4574\") pod \"payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:54.570541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.565067 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:54.642559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.642022 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4574\" (UniqueName: \"kubernetes.io/projected/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-kube-api-access-d4574\") pod \"payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:54.847716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:54.847620 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:55.359432 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:00:55.359396 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fbe00b_2b6a_4550_9e51_9be201cd82c5.slice/crio-901f454bc11eeac927c163c247820b6aa79331a31369b4102bde5da9fdf97d78 WatchSource:0}: Error finding container 901f454bc11eeac927c163c247820b6aa79331a31369b4102bde5da9fdf97d78: Status 404 returned error can't find the container with id 901f454bc11eeac927c163c247820b6aa79331a31369b4102bde5da9fdf97d78 Oct 30 01:00:55.363330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:55.360805 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx"] Oct 30 01:00:55.944169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:55.944138 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" event={"ID":"c8fbe00b-2b6a-4550-9e51-9be201cd82c5","Type":"ContainerStarted","Data":"60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b"} Oct 30 01:00:55.944377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:55.944193 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" event={"ID":"c8fbe00b-2b6a-4550-9e51-9be201cd82c5","Type":"ContainerStarted","Data":"901f454bc11eeac927c163c247820b6aa79331a31369b4102bde5da9fdf97d78"} Oct 30 01:00:55.944377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:55.944278 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:00:55.999542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:55.996537 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" podStartSLOduration=1.996495723 podStartE2EDuration="1.996495723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:55.993845314 +0000 UTC m=+1340.937018519" watchObservedRunningTime="2026-10-30 01:00:55.996495723 +0000 UTC m=+1340.939668916" Oct 30 01:00:56.058570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:56.052739 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk"] Oct 30 01:00:56.073190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:56.071581 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b44830f-nw5zk"] Oct 30 01:00:57.193400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:00:57.193367 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8078c6c1-0bc7-40ac-91d9-d23f84f4d5de" path="/var/lib/kubelet/pods/8078c6c1-0bc7-40ac-91d9-d23f84f4d5de/volumes" Oct 30 01:01:02.919488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:02.918968 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:01:02.949031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:02.949000 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:01:04.419794 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:01:04.419745 4905 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.235s" Oct 30 01:01:04.625668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.625608 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:04.694664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.694564 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p"] Oct 30 01:01:04.711532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.708728 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:04.760557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.753747 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:04.807073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.806836 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:04.862545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.861759 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p"] Oct 30 01:01:04.862545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.861898 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:04.909977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.909948 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b814fb-openshift-default-dockercfg-mp8rv\"" Oct 30 01:01:04.915085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.912623 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.002608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:04.977984 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.003091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.003056 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.003365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.003341 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.003501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.003483 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ncm\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-kube-api-access-z9ncm\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.003671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.003652 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.003854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.003838 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33cb223-2191-4064-87d5-0ce005175455-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.004161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.004139 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.004314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.004298 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.004543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.004504 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33cb223-2191-4064-87d5-0ce005175455-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.113103 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.113062 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.115064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.115034 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.170292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.170252 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ncm\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-kube-api-access-z9ncm\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.171018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.114426 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.171797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.171208 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.172072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.172041 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33cb223-2191-4064-87d5-0ce005175455-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.194545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.176171 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.194545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.176232 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.194545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.176305 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33cb223-2191-4064-87d5-0ce005175455-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.194545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.176539 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.224847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.224795 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.225353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.225329 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.226165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.226141 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33cb223-2191-4064-87d5-0ce005175455-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.242545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.238926 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.261354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.261273 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ncm\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-kube-api-access-z9ncm\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.306224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.306193 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.312194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.312162 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.321147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.321111 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33cb223-2191-4064-87d5-0ce005175455-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.623595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.622834 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:05.673544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.654667 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.686080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.680789 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:01:05.686080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.681365 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:01:05.717605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.716627 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 01:01:05.719107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.719071 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 01:01:05.719684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.719658 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:01:05.818554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.815667 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.818554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.816545 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:01:05.939605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.914060 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:01:05.939605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.921608 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.939605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:05.922225 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:01:06.035393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.010457 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.141433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.121829 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:01:06.672342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.672300 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.711398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.697933 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:01:06.711398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.681209 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:01:06.729549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.721488 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.756835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.756792 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.870802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.870767 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:01:06.904402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.894948 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 01:01:06.932313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:06.932190 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.075353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.075125 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.263305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.077797 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:01:07.263305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.100623 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:01:07.263305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.140996 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:01:07.263305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.142431 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:07.263305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.254089 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:07.263305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.254150 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:07.702079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.701988 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739331 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p"] Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739449 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" event={"ID":"a33cb223-2191-4064-87d5-0ce005175455","Type":"ContainerStarted","Data":"2c6a7a98803ec5d9befa88b389e338fb5feb02aa88699436ad6e12a706d80a84"} Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739579 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739660 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739741 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739804 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739878 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.739949 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.740017 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.750926 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.803055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.801454 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.869620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.868985 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:01:07.885411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.885366 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:01:07.939581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:07.932875 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:08.178359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.178329 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:01:08.178761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.178734 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:01:08.179016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.178994 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:01:08.179227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.179214 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:01:08.179444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.179430 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:01:08.297105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.297063 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:01:08.301247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.301222 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:01:08.315027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.314997 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:01:08.498425 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.498393 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" event={"ID":"a33cb223-2191-4064-87d5-0ce005175455","Type":"ContainerStarted","Data":"03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163"} Oct 30 01:01:08.672545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.664934 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:08.791164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:08.791049 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podStartSLOduration=5.791034082 podStartE2EDuration="5.791034082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:08.787734316 +0000 UTC m=+1353.730907540" watchObservedRunningTime="2026-10-30 01:01:08.791034082 +0000 UTC m=+1353.734207274" Oct 30 01:01:09.662544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:09.662052 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:10.045875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:10.045842 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc"] Oct 30 01:01:10.051808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:10.051779 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e9a128-j6hqc"] Oct 30 01:01:10.527542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:10.520193 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:10.527542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:10.524467 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:01:11.076555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:11.076367 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6"] Oct 30 01:01:11.183583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:11.175952 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0976d365-l6ff6"] Oct 30 01:01:11.215362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:11.215328 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3a486f-8f39-4cf0-8113-aacea94f695b" path="/var/lib/kubelet/pods/6d3a486f-8f39-4cf0-8113-aacea94f695b/volumes" Oct 30 01:01:11.215898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:11.215882 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8211c053-91f4-4864-8883-2d08931aa5c9" path="/var/lib/kubelet/pods/8211c053-91f4-4864-8883-2d08931aa5c9/volumes" Oct 30 01:01:12.417331 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:01:12.417289 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:01:13.205705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.205664 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m"] Oct 30 01:01:13.216402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.216369 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:13.223618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.221638 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m"] Oct 30 01:01:13.314540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.283913 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b814fb-dockercfg-vsdwd\"" Oct 30 01:01:13.314540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.284657 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b814fb\"" Oct 30 01:01:13.503713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.483718 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqz5d\" (UniqueName: \"kubernetes.io/projected/84b515ec-60fb-4b04-b97c-57141b253fb3-kube-api-access-bqz5d\") pod \"payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:13.503713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.483797 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b515ec-60fb-4b04-b97c-57141b253fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:13.618222 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.617442 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqz5d\" (UniqueName: \"kubernetes.io/projected/84b515ec-60fb-4b04-b97c-57141b253fb3-kube-api-access-bqz5d\") pod \"payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:13.618222 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.617598 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b515ec-60fb-4b04-b97c-57141b253fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:13.619180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.619026 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b515ec-60fb-4b04-b97c-57141b253fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:13.663228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.663139 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqz5d\" (UniqueName: \"kubernetes.io/projected/84b515ec-60fb-4b04-b97c-57141b253fb3-kube-api-access-bqz5d\") pod \"payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:13.958685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:13.958603 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:14.037800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:14.037740 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj"] Oct 30 01:01:14.047154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:14.046944 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1253324e-txqnj"] Oct 30 01:01:14.308558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:14.301804 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m"] Oct 30 01:01:14.754090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:14.754043 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" event={"ID":"84b515ec-60fb-4b04-b97c-57141b253fb3","Type":"ContainerStarted","Data":"dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58"} Oct 30 01:01:14.754511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:14.754107 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" event={"ID":"84b515ec-60fb-4b04-b97c-57141b253fb3","Type":"ContainerStarted","Data":"6d9d23af3fdf4a3ec59c5f6d2db9ebc48be850a18875da57e36b5e7c35c8ce97"} Oct 30 01:01:14.754511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:14.754136 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:14.799104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:14.799056 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" podStartSLOduration=1.799043609 podStartE2EDuration="1.799043609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:14.794732035 +0000 UTC m=+1359.737905226" watchObservedRunningTime="2026-10-30 01:01:14.799043609 +0000 UTC m=+1359.742216826" Oct 30 01:01:15.194763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:15.194671 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c35ed98-93b1-4adc-8e86-565bce5af6ca" path="/var/lib/kubelet/pods/3c35ed98-93b1-4adc-8e86-565bce5af6ca/volumes" Oct 30 01:01:15.802716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:15.802676 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:15.803244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:15.802724 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:16.790765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:16.790726 4905 generic.go:356] "Generic (PLEG): container finished" podID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerID="b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc" exitCode=0 Oct 30 01:01:16.790993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:16.790805 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" event={"ID":"90e4567a-0444-4693-a2f0-c832eaf6e6ff","Type":"ContainerDied","Data":"b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc"} Oct 30 01:01:16.791308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:16.791291 4905 scope.go:122] "RemoveContainer" containerID="b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc" Oct 30 01:01:16.793128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:16.793092 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:16.793262 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:16.793180 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:16.793262 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:16.793229 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:17.802910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:17.802865 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" event={"ID":"90e4567a-0444-4693-a2f0-c832eaf6e6ff","Type":"ContainerStarted","Data":"ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032"} Oct 30 01:01:17.817555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:17.817486 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:01:17.817826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:17.817805 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:01:17.817871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:17.817855 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:01:18.817445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:18.817409 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:01:18.817876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:18.817468 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:01:19.080602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:19.079667 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4"] Oct 30 01:01:19.092310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:19.089807 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037a9438-jdsl4"] Oct 30 01:01:19.192871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:19.192837 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d045905c-65eb-4e9b-92e8-121efb05f49f" path="/var/lib/kubelet/pods/d045905c-65eb-4e9b-92e8-121efb05f49f/volumes" Oct 30 01:01:19.817890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:19.817858 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:01:19.818340 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:19.817916 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:01:20.817637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:20.817593 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:01:20.817911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:20.817659 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:01:21.817924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:21.817888 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:01:21.818337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:21.817955 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:01:22.057706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:22.057653 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw"] Oct 30 01:01:22.063186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:22.063157 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00bc1f61-vpgnw"] Oct 30 01:01:22.818199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:22.818158 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:01:22.818738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:22.818232 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:01:23.071488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:23.071384 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7"] Oct 30 01:01:23.121619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:23.114443 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87470f9e-mxls7"] Oct 30 01:01:23.193808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:23.193775 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d806486c-d2b5-4e50-9610-4eba7e20d43d" path="/var/lib/kubelet/pods/d806486c-d2b5-4e50-9610-4eba7e20d43d/volumes" Oct 30 01:01:23.194306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:23.194292 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc21080-9ea1-4af3-b050-deba98eeec85" path="/var/lib/kubelet/pods/fcc21080-9ea1-4af3-b050-deba98eeec85/volumes" Oct 30 01:01:23.822164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:23.822135 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:01:23.822632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:23.822616 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:01:23.823395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:23.823377 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:01:25.781665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:25.781634 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:01:28.815134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.815104 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z"] Oct 30 01:01:28.819557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.819536 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:28.840155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.840123 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z"] Oct 30 01:01:28.849286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.845635 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:01:28.849286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.845650 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:01:28.854040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.854018 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:01:28.854176 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.854130 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:01:28.940588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.940550 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ks2\" (UniqueName: \"kubernetes.io/projected/999218ab-577e-4c0d-816f-12cf38c59666-kube-api-access-c9ks2\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:28.940767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.940705 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:28.940936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:28.940918 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.042151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.042120 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.042314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.042208 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ks2\" (UniqueName: \"kubernetes.io/projected/999218ab-577e-4c0d-816f-12cf38c59666-kube-api-access-c9ks2\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.042314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.042275 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.042968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.042945 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.044750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.044723 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.064207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.064180 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ks2\" (UniqueName: \"kubernetes.io/projected/999218ab-577e-4c0d-816f-12cf38c59666-kube-api-access-c9ks2\") pod \"maas-api-revoke-sub-f22510f0-rw45z\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.133927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.133862 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:29.519613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.489507 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z"] Oct 30 01:01:29.993946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.993844 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" event={"ID":"999218ab-577e-4c0d-816f-12cf38c59666","Type":"ContainerStarted","Data":"59f6c5301b45921783e583036ceb95f3d141ce52ab87eca629b6af48c593979b"} Oct 30 01:01:29.993946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:29.993905 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" event={"ID":"999218ab-577e-4c0d-816f-12cf38c59666","Type":"ContainerStarted","Data":"a8e7bc5a995ae3cea7a2baed33a4c235bcdd1a5c6d829c37975df8e1335c5cfd"} Oct 30 01:01:30.040599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:30.040475 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" podStartSLOduration=2.040458994 podStartE2EDuration="2.040458994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:30.035361181 +0000 UTC m=+1374.978534373" watchObservedRunningTime="2026-10-30 01:01:30.040458994 +0000 UTC m=+1374.983632184" Oct 30 01:01:31.005333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:31.005297 4905 generic.go:356] "Generic (PLEG): container finished" podID="999218ab-577e-4c0d-816f-12cf38c59666" containerID="59f6c5301b45921783e583036ceb95f3d141ce52ab87eca629b6af48c593979b" exitCode=0 Oct 30 01:01:31.005828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:31.005383 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" event={"ID":"999218ab-577e-4c0d-816f-12cf38c59666","Type":"ContainerDied","Data":"59f6c5301b45921783e583036ceb95f3d141ce52ab87eca629b6af48c593979b"} Oct 30 01:01:32.906081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:32.906001 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:33.000767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.000734 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-openshift-ca\") pod \"999218ab-577e-4c0d-816f-12cf38c59666\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " Oct 30 01:01:33.000938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.000826 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-xks-ca\") pod \"999218ab-577e-4c0d-816f-12cf38c59666\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " Oct 30 01:01:33.000997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.000984 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999218ab-577e-4c0d-816f-12cf38c59666-kube-api-access-c9ks2\" (UniqueName: \"kubernetes.io/projected/999218ab-577e-4c0d-816f-12cf38c59666-kube-api-access-c9ks2\") pod \"999218ab-577e-4c0d-816f-12cf38c59666\" (UID: \"999218ab-577e-4c0d-816f-12cf38c59666\") " Oct 30 01:01:33.001419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.001394 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-openshift-ca" pod "999218ab-577e-4c0d-816f-12cf38c59666" (UID: "999218ab-577e-4c0d-816f-12cf38c59666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:33.001697 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.001678 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:01:33.003748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.003721 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-xks-ca" pod "999218ab-577e-4c0d-816f-12cf38c59666" (UID: "999218ab-577e-4c0d-816f-12cf38c59666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:33.003850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.003796 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999218ab-577e-4c0d-816f-12cf38c59666-kube-api-access-c9ks2" pod "999218ab-577e-4c0d-816f-12cf38c59666" (UID: "999218ab-577e-4c0d-816f-12cf38c59666"). InnerVolumeSpecName "kube-api-access-c9ks2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:33.103238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.103105 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999218ab-577e-4c0d-816f-12cf38c59666-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:01:33.103238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.103134 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9ks2\" (UniqueName: \"kubernetes.io/projected/999218ab-577e-4c0d-816f-12cf38c59666-kube-api-access-c9ks2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:01:33.104418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.103896 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" event={"ID":"999218ab-577e-4c0d-816f-12cf38c59666","Type":"ContainerDied","Data":"a8e7bc5a995ae3cea7a2baed33a4c235bcdd1a5c6d829c37975df8e1335c5cfd"} Oct 30 01:01:33.104418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.103951 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8e7bc5a995ae3cea7a2baed33a4c235bcdd1a5c6d829c37975df8e1335c5cfd" Oct 30 01:01:33.104418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:33.104016 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z" Oct 30 01:01:34.059551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:34.053350 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8"] Oct 30 01:01:34.095812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:34.095776 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328f54ae-wdft8"] Oct 30 01:01:35.377674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:35.358185 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e04f1f-b10b-49cc-a71b-77c7fee573d0" path="/var/lib/kubelet/pods/75e04f1f-b10b-49cc-a71b-77c7fee573d0/volumes" Oct 30 01:01:38.036137 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.036088 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh"] Oct 30 01:01:38.099150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.099110 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98984103-9j2mh"] Oct 30 01:01:38.286382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.286287 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6"] Oct 30 01:01:38.295612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.292042 4905 cpu_manager.go:414] "containerMap: removing container" podUID="999218ab-577e-4c0d-816f-12cf38c59666" containerName="revoke-keys" Oct 30 01:01:38.295612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.292066 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999218ab-577e-4c0d-816f-12cf38c59666" containerName="revoke-keys" Oct 30 01:01:38.295612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.292256 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="999218ab-577e-4c0d-816f-12cf38c59666" containerName="revoke-keys" Oct 30 01:01:38.318743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.318706 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6"] Oct 30 01:01:38.318950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.318825 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.361638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.349691 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-283eb6-openshift-default-dockercfg-kw2jr\"" Oct 30 01:01:38.411060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411029 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-certs\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411073 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istiod-ca-cert\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411241 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-credential-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411302 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtq4x\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-kube-api-access-vtq4x\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411350 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411487 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-data\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411630 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-token\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411800 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-podinfo\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.411888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.411827 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-envoy\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.517852 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-certs\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.517907 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istiod-ca-cert\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.517959 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-credential-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.518009 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtq4x\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-kube-api-access-vtq4x\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.518053 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.518100 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-data\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.518187 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-token\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.518359 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-podinfo\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.518398 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-envoy\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.521342 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-certs\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.521984 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istiod-ca-cert\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.522873 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-credential-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.523142 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-data\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.540621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.523390 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.646595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.645472 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-podinfo\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.647493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.647428 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtq4x\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-kube-api-access-vtq4x\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.715559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.683366 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-token\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.715559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.683726 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-envoy\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:38.716382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:38.716349 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:39.207713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.206460 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63b0a10-feb0-42c0-87b6-33bece8d721d" path="/var/lib/kubelet/pods/c63b0a10-feb0-42c0-87b6-33bece8d721d/volumes" Oct 30 01:01:39.424644 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:01:39.424611 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc90b0e_4f0c_4104_8860_849aaf6772dc.slice/crio-89bca070714d653e122dd426c607b42c4eadd59389f724efdaabf39173b6b6ad WatchSource:0}: Error finding container 89bca070714d653e122dd426c607b42c4eadd59389f724efdaabf39173b6b6ad: Status 404 returned error can't find the container with id 89bca070714d653e122dd426c607b42c4eadd59389f724efdaabf39173b6b6ad Oct 30 01:01:39.428125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.428081 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:39.428238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.428172 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:39.428300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.428237 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:39.443758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.443732 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6"] Oct 30 01:01:39.877832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.877792 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" event={"ID":"0cc90b0e-4f0c-4104-8860-849aaf6772dc","Type":"ContainerStarted","Data":"f090cbb7561550b044e6daf15c65c84321903c3b4086737276882711a88e8a3c"} Oct 30 01:01:39.878030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.877858 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" event={"ID":"0cc90b0e-4f0c-4104-8860-849aaf6772dc","Type":"ContainerStarted","Data":"89bca070714d653e122dd426c607b42c4eadd59389f724efdaabf39173b6b6ad"} Oct 30 01:01:39.942592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:39.940552 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podStartSLOduration=1.9405322310000002 podStartE2EDuration="1.940532231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:39.938415993 +0000 UTC m=+1384.881589185" watchObservedRunningTime="2026-10-30 01:01:39.940532231 +0000 UTC m=+1384.883705418" Oct 30 01:01:40.032935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.032906 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr"] Oct 30 01:01:40.041539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.041491 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d94875-dg5lr"] Oct 30 01:01:40.213020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.212927 4905 scope.go:122] "RemoveContainer" containerID="b4d01791fd2fb5e5058ea4491615e8a7737956bcde441ff58f5737c784d33b77" Oct 30 01:01:40.242179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.242150 4905 scope.go:122] "RemoveContainer" containerID="b77e3ef59d754da11577b4ff949aeb1b439122b76020ae623085cee4cc131368" Oct 30 01:01:40.738662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.738627 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:40.747741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.747713 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:40.891901 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.891860 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:40.893005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:40.892980 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:01:41.198373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:41.198273 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5" path="/var/lib/kubelet/pods/b5048b5e-7d50-4aeb-9de1-d0f2d8a8f4b5/volumes" Oct 30 01:01:42.069792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:42.069755 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl"] Oct 30 01:01:42.080835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:42.080810 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9349751a-w22gl"] Oct 30 01:01:42.754579 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:01:42.747792 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:01:43.195011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:43.194934 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b12fda0-42b7-49a9-86db-97fb4d3a2b0e" path="/var/lib/kubelet/pods/2b12fda0-42b7-49a9-86db-97fb4d3a2b0e/volumes" Oct 30 01:01:44.117679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.117644 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q"] Oct 30 01:01:44.159636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.154645 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q"] Oct 30 01:01:44.159636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.155055 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:44.242006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.200790 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-283eb6\"" Oct 30 01:01:44.242006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.200890 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-283eb6-dockercfg-mpgjd\"" Oct 30 01:01:44.309550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.302007 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca97f62-eaab-4526-8241-087fcb9ba59d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:44.309550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.302221 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8z5h\" (UniqueName: \"kubernetes.io/projected/7ca97f62-eaab-4526-8241-087fcb9ba59d-kube-api-access-n8z5h\") pod \"payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:44.446610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.442672 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8z5h\" (UniqueName: \"kubernetes.io/projected/7ca97f62-eaab-4526-8241-087fcb9ba59d-kube-api-access-n8z5h\") pod \"payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:44.446610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.443024 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca97f62-eaab-4526-8241-087fcb9ba59d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:44.532746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.525745 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca97f62-eaab-4526-8241-087fcb9ba59d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:44.611256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.605343 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8z5h\" (UniqueName: \"kubernetes.io/projected/7ca97f62-eaab-4526-8241-087fcb9ba59d-kube-api-access-n8z5h\") pod \"payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:44.622092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.622059 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz"] Oct 30 01:01:44.650174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.650111 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz"] Oct 30 01:01:44.650629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.650574 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.055483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.769887 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.055483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.770021 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnbb\" (UniqueName: \"kubernetes.io/projected/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-kube-api-access-ljnbb\") pod \"payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.055483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.817481 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:45.055483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.873834 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.055483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.873936 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnbb\" (UniqueName: \"kubernetes.io/projected/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-kube-api-access-ljnbb\") pod \"payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.055483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.875323 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.055483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:44.989086 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnbb\" (UniqueName: \"kubernetes.io/projected/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-kube-api-access-ljnbb\") pod \"payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.283470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:45.283439 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:45.401895 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:01:45.401862 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca97f62_eaab_4526_8241_087fcb9ba59d.slice/crio-5a546f235fef44ab1bcd91fea5f4280f5410b0a252249688a7100a8dc14a7e41 WatchSource:0}: Error finding container 5a546f235fef44ab1bcd91fea5f4280f5410b0a252249688a7100a8dc14a7e41: Status 404 returned error can't find the container with id 5a546f235fef44ab1bcd91fea5f4280f5410b0a252249688a7100a8dc14a7e41 Oct 30 01:01:45.406984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:45.406960 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q"] Oct 30 01:01:45.588442 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:01:45.588410 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dd91e5_c8fb_4c54_9ba0_172b6cef3028.slice/crio-ad0766e69dbd6e27939d8f47a1e0ed0b9fc7f032b15ba3bbe6b81dbb079a9e0e WatchSource:0}: Error finding container ad0766e69dbd6e27939d8f47a1e0ed0b9fc7f032b15ba3bbe6b81dbb079a9e0e: Status 404 returned error can't find the container with id ad0766e69dbd6e27939d8f47a1e0ed0b9fc7f032b15ba3bbe6b81dbb079a9e0e Oct 30 01:01:45.590828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:45.590732 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz"] Oct 30 01:01:45.600949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:45.600920 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:45.601129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:45.601099 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:46.052050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.052007 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" event={"ID":"7ca97f62-eaab-4526-8241-087fcb9ba59d","Type":"ContainerStarted","Data":"0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87"} Oct 30 01:01:46.052283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.052119 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:46.052283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.052146 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" event={"ID":"7ca97f62-eaab-4526-8241-087fcb9ba59d","Type":"ContainerStarted","Data":"5a546f235fef44ab1bcd91fea5f4280f5410b0a252249688a7100a8dc14a7e41"} Oct 30 01:01:46.054161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.054134 4905 generic.go:356] "Generic (PLEG): container finished" podID="34b8803c-bbf8-434b-91de-024c924643cc" containerID="af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d" exitCode=0 Oct 30 01:01:46.054273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.054208 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" event={"ID":"34b8803c-bbf8-434b-91de-024c924643cc","Type":"ContainerDied","Data":"af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d"} Oct 30 01:01:46.054792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.054685 4905 scope.go:122] "RemoveContainer" containerID="af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d" Oct 30 01:01:46.056440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.056419 4905 generic.go:356] "Generic (PLEG): container finished" podID="edc9348d-15c2-410a-bcd7-3781c7888192" containerID="5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0" exitCode=0 Oct 30 01:01:46.056561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.056494 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" event={"ID":"edc9348d-15c2-410a-bcd7-3781c7888192","Type":"ContainerDied","Data":"5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0"} Oct 30 01:01:46.056801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.056769 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:46.056879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.056855 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:46.056933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.056898 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:46.056933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.056916 4905 scope.go:122] "RemoveContainer" containerID="5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0" Oct 30 01:01:46.058779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.058747 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:46.058868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.058830 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:46.058914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.058874 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:46.059622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.059600 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" event={"ID":"64dd91e5-c8fb-4c54-9ba0-172b6cef3028","Type":"ContainerStarted","Data":"ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142"} Oct 30 01:01:46.059784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.059764 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" event={"ID":"64dd91e5-c8fb-4c54-9ba0-172b6cef3028","Type":"ContainerStarted","Data":"ad0766e69dbd6e27939d8f47a1e0ed0b9fc7f032b15ba3bbe6b81dbb079a9e0e"} Oct 30 01:01:46.059872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.059798 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:46.101146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.101095 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" podStartSLOduration=2.101077266 podStartE2EDuration="2.101077266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:46.092474636 +0000 UTC m=+1391.035647828" watchObservedRunningTime="2026-10-30 01:01:46.101077266 +0000 UTC m=+1391.044250460" Oct 30 01:01:46.190628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:46.190577 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" podStartSLOduration=2.190558106 podStartE2EDuration="2.190558106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:46.187668331 +0000 UTC m=+1391.130841523" watchObservedRunningTime="2026-10-30 01:01:46.190558106 +0000 UTC m=+1391.133731296" Oct 30 01:01:47.151465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.151407 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" event={"ID":"edc9348d-15c2-410a-bcd7-3781c7888192","Type":"ContainerStarted","Data":"ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3"} Oct 30 01:01:47.184615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.181448 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" event={"ID":"34b8803c-bbf8-434b-91de-024c924643cc","Type":"ContainerStarted","Data":"6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9"} Oct 30 01:01:47.229765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.229729 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:47.230099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.230069 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:47.313549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.308312 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:01:47.514622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.514585 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:01:47.516143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.516106 4905 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-mljpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:01:47.516257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:47.516164 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:01:48.097189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:48.096168 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk"] Oct 30 01:01:48.097189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:48.096238 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c098b524-z57wk"] Oct 30 01:01:48.218190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:48.218159 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:48.218633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:48.218206 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:48.522544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:48.519941 4905 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-mljpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:01:48.522544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:48.519999 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:01:49.202279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.202240 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f77110-832f-4cc5-bff7-ae34fbfc719a" path="/var/lib/kubelet/pods/66f77110-832f-4cc5-bff7-ae34fbfc719a/volumes" Oct 30 01:01:49.218536 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.218491 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:49.218970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.218564 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:49.569546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.566389 4905 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-mljpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:01:49.569546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.566466 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:01:49.578879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.578839 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944"] Oct 30 01:01:49.618086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.618059 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.619747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.617957 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944"] Oct 30 01:01:49.654545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.649180 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-30c405-openshift-default-dockercfg-4vp7g\"" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691320 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-podinfo\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691382 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jg7\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-kube-api-access-s6jg7\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691490 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-envoy\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691613 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-socket\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691748 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-credential-socket\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691815 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdfb72-abd3-4976-930f-b96e5887ccef-istiod-ca-cert\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691857 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-token\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691928 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-certs\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.752541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.691957 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-data\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.800681 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-podinfo\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.800751 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jg7\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-kube-api-access-s6jg7\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.800857 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-envoy\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.800962 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-socket\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.801091 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-credential-socket\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.801170 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdfb72-abd3-4976-930f-b96e5887ccef-istiod-ca-cert\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.801211 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-token\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.801279 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-certs\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.801307 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-data\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.812815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.801983 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-data\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.838839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.813375 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-credential-socket\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.838839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.815398 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdfb72-abd3-4976-930f-b96e5887ccef-istiod-ca-cert\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.838839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.815815 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-certs\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.838839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.816212 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-socket\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.893565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.884975 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-envoy\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.896405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.896345 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jg7\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-kube-api-access-s6jg7\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.937541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.935735 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-token\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:49.941544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:49.938882 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-podinfo\") pod \"e2e-api-b-30c405-openshift-default-64ddb7f597-sf944\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:50.123441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.123312 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:50.218956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.218922 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:50.219628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.219467 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:50.515581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.514879 4905 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-mljpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:01:50.515581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.514936 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:01:50.723546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.721712 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944"] Oct 30 01:01:50.723546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.722761 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:50.723546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.722860 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:50.723546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:50.722902 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:01:51.218033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.218005 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:51.218308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.218048 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:51.260557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.255630 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:01:51.323176 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.323129 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" event={"ID":"4bcdfb72-abd3-4976-930f-b96e5887ccef","Type":"ContainerStarted","Data":"b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680"} Oct 30 01:01:51.323484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.323463 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" event={"ID":"4bcdfb72-abd3-4976-930f-b96e5887ccef","Type":"ContainerStarted","Data":"0604b0605a558923a4b1e035fd3ce72718d1a4b7dc88ad58af4c03f05e1ee3fa"} Oct 30 01:01:51.500568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.499126 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podStartSLOduration=2.499106091 podStartE2EDuration="2.499106091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:51.49404803 +0000 UTC m=+1396.437221227" watchObservedRunningTime="2026-10-30 01:01:51.499106091 +0000 UTC m=+1396.442279284" Oct 30 01:01:51.514907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.514872 4905 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-mljpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:01:51.515066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:51.514938 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:01:52.148545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.140699 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:52.182613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.175860 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:52.218322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.218281 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:52.218587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.218338 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:52.381544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.380820 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:52.390560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.390343 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:01:52.515319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.514981 4905 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-mljpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:01:52.515319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:52.515038 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:01:53.218852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:53.218818 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:01:53.219040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:53.218870 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:01:53.515500 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:53.515460 4905 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-mljpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:01:53.515940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:53.515552 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:01:54.273552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:54.273294 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:01:54.277541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:54.275097 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:01:54.289540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:54.281504 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:01:54.521275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:54.521242 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:01:54.522407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:54.522391 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:01:55.634831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.634789 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6"] Oct 30 01:01:55.639875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.639849 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:55.667815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.667782 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-30c405-dockercfg-fmpcs\"" Oct 30 01:01:55.668698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.668675 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-30c405\"" Oct 30 01:01:55.675592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.672894 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6"] Oct 30 01:01:55.846560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.843588 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1330a7f0-0608-435a-b516-3a6be58d692b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:55.846560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.843700 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqgst\" (UniqueName: \"kubernetes.io/projected/1330a7f0-0608-435a-b516-3a6be58d692b-kube-api-access-kqgst\") pod \"payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:55.887251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.887166 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh"] Oct 30 01:01:55.959544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.958976 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1330a7f0-0608-435a-b516-3a6be58d692b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:55.959544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.959158 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgst\" (UniqueName: \"kubernetes.io/projected/1330a7f0-0608-435a-b516-3a6be58d692b-kube-api-access-kqgst\") pod \"payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:55.960435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:55.960401 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1330a7f0-0608-435a-b516-3a6be58d692b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:56.055550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.044801 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.114802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.114298 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqmv\" (UniqueName: \"kubernetes.io/projected/f4641871-a640-4404-adec-c62f1a46acb4-kube-api-access-5cqmv\") pod \"payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.114802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.114397 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4641871-a640-4404-adec-c62f1a46acb4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.228544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.222028 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqmv\" (UniqueName: \"kubernetes.io/projected/f4641871-a640-4404-adec-c62f1a46acb4-kube-api-access-5cqmv\") pod \"payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.228544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.222109 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4641871-a640-4404-adec-c62f1a46acb4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.228544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.223227 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4641871-a640-4404-adec-c62f1a46acb4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.261842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.255972 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgst\" (UniqueName: \"kubernetes.io/projected/1330a7f0-0608-435a-b516-3a6be58d692b-kube-api-access-kqgst\") pod \"payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:56.261842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.256339 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqmv\" (UniqueName: \"kubernetes.io/projected/f4641871-a640-4404-adec-c62f1a46acb4-kube-api-access-5cqmv\") pod \"payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.261842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.259950 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh"] Oct 30 01:01:56.266461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.266311 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:56.554487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.554449 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:56.588157 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:01:56.588123 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1330a7f0_0608_435a_b516_3a6be58d692b.slice/crio-4e9769fb959b47f30f12704784392c4f6135fd54971a2e2f21f0f70bd60ba647 WatchSource:0}: Error finding container 4e9769fb959b47f30f12704784392c4f6135fd54971a2e2f21f0f70bd60ba647: Status 404 returned error can't find the container with id 4e9769fb959b47f30f12704784392c4f6135fd54971a2e2f21f0f70bd60ba647 Oct 30 01:01:56.591453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.591425 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6"] Oct 30 01:01:56.869874 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:01:56.869841 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4641871_a640_4404_adec_c62f1a46acb4.slice/crio-98228d2fb0be75ee6a926025104060bcfed378675041c3a2c322f3dff572561c WatchSource:0}: Error finding container 98228d2fb0be75ee6a926025104060bcfed378675041c3a2c322f3dff572561c: Status 404 returned error can't find the container with id 98228d2fb0be75ee6a926025104060bcfed378675041c3a2c322f3dff572561c Oct 30 01:01:56.872144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:56.872096 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh"] Oct 30 01:01:57.040709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.040671 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx"] Oct 30 01:01:57.045857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.045830 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e1b12e-fx9cx"] Oct 30 01:01:57.194174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.194092 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aa12f4-a34b-4d0a-b34b-f5bea87b554e" path="/var/lib/kubelet/pods/58aa12f4-a34b-4d0a-b34b-f5bea87b554e/volumes" Oct 30 01:01:57.194716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.194699 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:01:57.194793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.194770 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:01:57.570292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.570252 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" event={"ID":"1330a7f0-0608-435a-b516-3a6be58d692b","Type":"ContainerStarted","Data":"3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae"} Oct 30 01:01:57.570510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.570322 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" event={"ID":"1330a7f0-0608-435a-b516-3a6be58d692b","Type":"ContainerStarted","Data":"4e9769fb959b47f30f12704784392c4f6135fd54971a2e2f21f0f70bd60ba647"} Oct 30 01:01:57.570510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.570390 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:01:57.572638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.572613 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" event={"ID":"f4641871-a640-4404-adec-c62f1a46acb4","Type":"ContainerStarted","Data":"50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39"} Oct 30 01:01:57.572755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.572659 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" event={"ID":"f4641871-a640-4404-adec-c62f1a46acb4","Type":"ContainerStarted","Data":"98228d2fb0be75ee6a926025104060bcfed378675041c3a2c322f3dff572561c"} Oct 30 01:01:57.572755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.572678 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:01:57.612459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.612412 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" podStartSLOduration=2.612399218 podStartE2EDuration="2.612399218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:57.610846907 +0000 UTC m=+1402.554020098" watchObservedRunningTime="2026-10-30 01:01:57.612399218 +0000 UTC m=+1402.555572409" Oct 30 01:01:57.655916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:01:57.655859 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" podStartSLOduration=2.655844563 podStartE2EDuration="2.655844563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:57.652307146 +0000 UTC m=+1402.595480338" watchObservedRunningTime="2026-10-30 01:01:57.655844563 +0000 UTC m=+1402.599017776" Oct 30 01:02:04.058633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:04.058595 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl"] Oct 30 01:02:04.098992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:04.098950 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e06912-5bnkl"] Oct 30 01:02:04.149639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:04.149569 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn"] Oct 30 01:02:04.192610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:04.192566 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9022e55a-xb6hn"] Oct 30 01:02:05.233581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.224297 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba548d1-84d4-42ea-b1ce-4def93eadc01" path="/var/lib/kubelet/pods/3ba548d1-84d4-42ea-b1ce-4def93eadc01/volumes" Oct 30 01:02:05.233581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.225044 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61bef333-81be-4014-9e2e-165b1ce2b00c" path="/var/lib/kubelet/pods/61bef333-81be-4014-9e2e-165b1ce2b00c/volumes" Oct 30 01:02:05.759294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.759251 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9"] Oct 30 01:02:05.765601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.765574 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.791906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.791878 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:02:05.792145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.792131 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:02:05.792284 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.792272 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:02:05.797710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.797657 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:02:05.806684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.804299 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9"] Oct 30 01:02:05.894231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.894192 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.894407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.894339 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztxs\" (UniqueName: \"kubernetes.io/projected/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-kube-api-access-8ztxs\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.894572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.894554 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.995895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.995861 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.996079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.996064 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.996206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.996192 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztxs\" (UniqueName: \"kubernetes.io/projected/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-kube-api-access-8ztxs\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.996682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.996655 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:05.998668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:05.998638 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:06.018326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:06.018273 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztxs\" (UniqueName: \"kubernetes.io/projected/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-kube-api-access-8ztxs\") pod \"maas-api-revoke-sub-ca7be503-rjsk9\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:06.090862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:06.090834 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:06.396209 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:02:06.396177 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5f4a08_94c3_49dc_9cf4_8dc0b29e88ce.slice/crio-4299b755ef65e59c4809fd4ec9138c57db99e299d2c3f3e0dbba4c1f53a0a692 WatchSource:0}: Error finding container 4299b755ef65e59c4809fd4ec9138c57db99e299d2c3f3e0dbba4c1f53a0a692: Status 404 returned error can't find the container with id 4299b755ef65e59c4809fd4ec9138c57db99e299d2c3f3e0dbba4c1f53a0a692 Oct 30 01:02:06.398917 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:06.398889 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9"] Oct 30 01:02:06.810913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:06.810881 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" event={"ID":"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce","Type":"ContainerStarted","Data":"dc4fce2fad9b903a0f004a3403a4e65ccda78532d13b5145d83d5ca7d294c214"} Oct 30 01:02:06.811051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:06.810934 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" event={"ID":"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce","Type":"ContainerStarted","Data":"4299b755ef65e59c4809fd4ec9138c57db99e299d2c3f3e0dbba4c1f53a0a692"} Oct 30 01:02:06.862976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:06.862932 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" podStartSLOduration=1.862919276 podStartE2EDuration="1.862919276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:06.856417394 +0000 UTC m=+1411.799590586" watchObservedRunningTime="2026-10-30 01:02:06.862919276 +0000 UTC m=+1411.806092465" Oct 30 01:02:07.822778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:07.822743 4905 generic.go:356] "Generic (PLEG): container finished" podID="8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" containerID="dc4fce2fad9b903a0f004a3403a4e65ccda78532d13b5145d83d5ca7d294c214" exitCode=0 Oct 30 01:02:07.823191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:07.822835 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" event={"ID":"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce","Type":"ContainerDied","Data":"dc4fce2fad9b903a0f004a3403a4e65ccda78532d13b5145d83d5ca7d294c214"} Oct 30 01:02:08.584298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:08.584269 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:02:08.584477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:08.584335 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:02:09.145588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.133851 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:09.182631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.182586 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-openshift-ca\") pod \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " Oct 30 01:02:09.182794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.182671 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-xks-ca\") pod \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " Oct 30 01:02:09.182996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.182977 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-kube-api-access-8ztxs\" (UniqueName: \"kubernetes.io/projected/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-kube-api-access-8ztxs\") pod \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\" (UID: \"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce\") " Oct 30 01:02:09.184255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.184191 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-openshift-ca" pod "8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" (UID: "8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:09.187684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.187657 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-xks-ca" pod "8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" (UID: "8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:09.187840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.187686 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-kube-api-access-8ztxs" pod "8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" (UID: "8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce"). InnerVolumeSpecName "kube-api-access-8ztxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:09.285783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.285750 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ztxs\" (UniqueName: \"kubernetes.io/projected/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-kube-api-access-8ztxs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:09.285783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.285783 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:09.286033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.285797 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:09.874815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.874784 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" Oct 30 01:02:09.874988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.874810 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9" event={"ID":"8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce","Type":"ContainerDied","Data":"4299b755ef65e59c4809fd4ec9138c57db99e299d2c3f3e0dbba4c1f53a0a692"} Oct 30 01:02:09.874988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:09.874868 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4299b755ef65e59c4809fd4ec9138c57db99e299d2c3f3e0dbba4c1f53a0a692" Oct 30 01:02:12.054939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:12.054901 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk"] Oct 30 01:02:12.149980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:12.149950 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b8365f-nkqtk"] Oct 30 01:02:12.493287 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:02:12.493234 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:02:13.033794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.033758 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5"] Oct 30 01:02:13.046645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.046610 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50bd9dff-7gbt5"] Oct 30 01:02:13.193666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.193594 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490cea2b-7c15-4d77-b163-6f0bbe49982d" path="/var/lib/kubelet/pods/490cea2b-7c15-4d77-b163-6f0bbe49982d/volumes" Oct 30 01:02:13.194298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.194270 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0ef1c2-a60c-4c24-a809-f02f8130a864" path="/var/lib/kubelet/pods/bd0ef1c2-a60c-4c24-a809-f02f8130a864/volumes" Oct 30 01:02:13.284265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.284232 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk"] Oct 30 01:02:13.285475 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.285434 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" containerName="revoke-keys" Oct 30 01:02:13.285475 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.285461 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" containerName="revoke-keys" Oct 30 01:02:13.285853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.285833 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" containerName="revoke-keys" Oct 30 01:02:13.296754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.296735 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.306893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.306863 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk"] Oct 30 01:02:13.313819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.313798 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ee0869-openshift-default-dockercfg-bf8ps\"" Oct 30 01:02:13.397746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.397673 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-data\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.397940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.397792 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-podinfo\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.397940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.397860 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-token\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.397940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.397935 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-credential-socket\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.398113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.397978 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-socket\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.398113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.398064 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd40be8-c62e-4831-8fe8-075ae7835375-istiod-ca-cert\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.398226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.398133 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-envoy\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.404721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.398283 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-certs\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.404721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.398330 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfrx\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-kube-api-access-vsfrx\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.519777 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-data\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.519887 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-podinfo\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.519941 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-token\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.520010 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-credential-socket\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.520051 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-socket\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.520130 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd40be8-c62e-4831-8fe8-075ae7835375-istiod-ca-cert\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.520195 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-envoy\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.520323 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-certs\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.523543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.520368 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfrx\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-kube-api-access-vsfrx\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.542570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.528321 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-credential-socket\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.542570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.528631 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-data\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.542570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.532614 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd40be8-c62e-4831-8fe8-075ae7835375-istiod-ca-cert\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.542570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.532851 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-socket\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.584550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.582098 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-certs\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.641243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.641203 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs"] Oct 30 01:02:13.653794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.653651 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs"] Oct 30 01:02:13.654042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.653763 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:13.654906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.654370 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-token\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.671682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.671146 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:02:13.671682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.671482 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:02:13.671682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.671596 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:02:13.671682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.671651 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:02:13.742543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.726947 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:13.811166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.810701 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:13.811166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.810798 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxkg\" (UniqueName: \"kubernetes.io/projected/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-kube-api-access-qjxkg\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:13.902391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.871852 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-envoy\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.915819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.915729 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-podinfo\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.965683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.937532 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:13.966829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.943425 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfrx\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-kube-api-access-vsfrx\") pod \"e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:13.966829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.966198 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:13.966829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.966266 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxkg\" (UniqueName: \"kubernetes.io/projected/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-kube-api-access-qjxkg\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:13.967402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:13.967329 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:14.034055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.020814 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:14.122547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.120778 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxkg\" (UniqueName: \"kubernetes.io/projected/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-kube-api-access-qjxkg\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-rmpxs\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:14.139693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.137146 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:14.230437 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.230202 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:14.516850 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:02:14.516806 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b4ed5f_cc87_42c3_b252_9a20545c8a4e.slice/crio-403015e4ed8447d6015b63175451a1c1655cbf0c262893545057f0459e7209e7 WatchSource:0}: Error finding container 403015e4ed8447d6015b63175451a1c1655cbf0c262893545057f0459e7209e7: Status 404 returned error can't find the container with id 403015e4ed8447d6015b63175451a1c1655cbf0c262893545057f0459e7209e7 Oct 30 01:02:14.520422 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.520396 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs"] Oct 30 01:02:14.569304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.569272 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk"] Oct 30 01:02:14.570252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.570207 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:14.570371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.570303 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:14.570371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:14.570344 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:15.078258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.077866 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" event={"ID":"72b4ed5f-cc87-42c3-b252-9a20545c8a4e","Type":"ContainerStarted","Data":"94c0d9d662cd2dae8dada90082ac35bac3322d320e1f7451064c32bcfa85e700"} Oct 30 01:02:15.078258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.077929 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" event={"ID":"72b4ed5f-cc87-42c3-b252-9a20545c8a4e","Type":"ContainerStarted","Data":"403015e4ed8447d6015b63175451a1c1655cbf0c262893545057f0459e7209e7"} Oct 30 01:02:15.080953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.080809 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" event={"ID":"5cd40be8-c62e-4831-8fe8-075ae7835375","Type":"ContainerStarted","Data":"39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0"} Oct 30 01:02:15.080953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.080851 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" event={"ID":"5cd40be8-c62e-4831-8fe8-075ae7835375","Type":"ContainerStarted","Data":"d36c90ead07ad818e2033c7caf961606d4cc7b339e655f742e0fa84ad3b3aa72"} Oct 30 01:02:15.121083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.121026 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" podStartSLOduration=2.120998952 podStartE2EDuration="2.120998952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:15.117964663 +0000 UTC m=+1420.061137855" watchObservedRunningTime="2026-10-30 01:02:15.120998952 +0000 UTC m=+1420.064172148" Oct 30 01:02:15.179669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.179615 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podStartSLOduration=2.179598805 podStartE2EDuration="2.179598805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:15.17480733 +0000 UTC m=+1420.117980522" watchObservedRunningTime="2026-10-30 01:02:15.179598805 +0000 UTC m=+1420.122771998" Oct 30 01:02:15.231510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.231447 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:15.237888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:15.237859 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:16.096679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:16.096644 4905 generic.go:356] "Generic (PLEG): container finished" podID="72b4ed5f-cc87-42c3-b252-9a20545c8a4e" containerID="94c0d9d662cd2dae8dada90082ac35bac3322d320e1f7451064c32bcfa85e700" exitCode=0 Oct 30 01:02:16.096871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:16.096731 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" event={"ID":"72b4ed5f-cc87-42c3-b252-9a20545c8a4e","Type":"ContainerDied","Data":"94c0d9d662cd2dae8dada90082ac35bac3322d320e1f7451064c32bcfa85e700"} Oct 30 01:02:16.097561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:16.097319 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:16.098334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:16.098311 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:02:17.029408 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.029370 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m"] Oct 30 01:02:17.045594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.045554 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab723a90-9557m"] Oct 30 01:02:17.196027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.195994 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086bd386-9177-4c16-835b-1adbef7553b4" path="/var/lib/kubelet/pods/086bd386-9177-4c16-835b-1adbef7553b4/volumes" Oct 30 01:02:17.334842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.334806 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:17.435321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.435278 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-xks-ca\") pod \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " Oct 30 01:02:17.435511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.435342 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-openshift-ca\") pod \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " Oct 30 01:02:17.435511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.435430 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-kube-api-access-qjxkg\" (UniqueName: \"kubernetes.io/projected/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-kube-api-access-qjxkg\") pod \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\" (UID: \"72b4ed5f-cc87-42c3-b252-9a20545c8a4e\") " Oct 30 01:02:17.437219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.437190 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-openshift-ca" pod "72b4ed5f-cc87-42c3-b252-9a20545c8a4e" (UID: "72b4ed5f-cc87-42c3-b252-9a20545c8a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:17.439658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.439628 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-xks-ca" pod "72b4ed5f-cc87-42c3-b252-9a20545c8a4e" (UID: "72b4ed5f-cc87-42c3-b252-9a20545c8a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:17.443816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.443790 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-kube-api-access-qjxkg" pod "72b4ed5f-cc87-42c3-b252-9a20545c8a4e" (UID: "72b4ed5f-cc87-42c3-b252-9a20545c8a4e"). InnerVolumeSpecName "kube-api-access-qjxkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:17.538728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.538686 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:17.538728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.538720 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:17.538728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:17.538734 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjxkg\" (UniqueName: \"kubernetes.io/projected/72b4ed5f-cc87-42c3-b252-9a20545c8a4e-kube-api-access-qjxkg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:18.207708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:18.206460 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" Oct 30 01:02:18.208200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:18.207940 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs" event={"ID":"72b4ed5f-cc87-42c3-b252-9a20545c8a4e","Type":"ContainerDied","Data":"403015e4ed8447d6015b63175451a1c1655cbf0c262893545057f0459e7209e7"} Oct 30 01:02:18.208200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:18.207987 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="403015e4ed8447d6015b63175451a1c1655cbf0c262893545057f0459e7209e7" Oct 30 01:02:19.225859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.225819 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw"] Oct 30 01:02:19.227424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.227358 4905 generic.go:356] "Generic (PLEG): container finished" podID="a33cb223-2191-4064-87d5-0ce005175455" containerID="03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163" exitCode=0 Oct 30 01:02:19.227642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.227622 4905 cpu_manager.go:414] "containerMap: removing container" podUID="72b4ed5f-cc87-42c3-b252-9a20545c8a4e" containerName="revoke-keys" Oct 30 01:02:19.227746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.227644 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b4ed5f-cc87-42c3-b252-9a20545c8a4e" containerName="revoke-keys" Oct 30 01:02:19.227904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.227887 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b4ed5f-cc87-42c3-b252-9a20545c8a4e" containerName="revoke-keys" Oct 30 01:02:19.231297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.231274 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" event={"ID":"a33cb223-2191-4064-87d5-0ce005175455","Type":"ContainerDied","Data":"03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163"} Oct 30 01:02:19.231561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.231506 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.231835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.231808 4905 scope.go:122] "RemoveContainer" containerID="03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163" Oct 30 01:02:19.235335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.235306 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:19.235551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.235510 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:19.235718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.235697 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:19.260543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.259922 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw"] Oct 30 01:02:19.290320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290285 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.290461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290391 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.290461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290444 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4acc83b2-74c4-4499-a96b-3eecead8223d-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.290611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290473 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.290611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290542 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5xc\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-kube-api-access-ds5xc\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.290741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290625 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.290741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290656 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.290854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.291019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.290998 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393018 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393145 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393204 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393284 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393250 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4acc83b2-74c4-4499-a96b-3eecead8223d-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393340 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393280 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393340 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393332 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5xc\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-kube-api-access-ds5xc\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393386 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393416 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393554 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393624 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393695 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393876 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.393984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.393963 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.394308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.394286 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4acc83b2-74c4-4499-a96b-3eecead8223d-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.402565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.396334 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.402565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.396835 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.434804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.434723 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5xc\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-kube-api-access-ds5xc\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.440454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.438983 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8"] Oct 30 01:02:19.445642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.445570 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.454926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.449969 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:19.472576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.464917 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8"] Oct 30 01:02:19.472576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.466984 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ee0869\"" Oct 30 01:02:19.473799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.473775 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ee0869-dockercfg-vzcr9\"" Oct 30 01:02:19.515545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.499275 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ae684f-bc2b-4cbf-9f1d-194630e85870-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:19.515545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.499339 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktlx\" (UniqueName: \"kubernetes.io/projected/90ae684f-bc2b-4cbf-9f1d-194630e85870-kube-api-access-fktlx\") pod \"payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:19.729543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.729394 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ae684f-bc2b-4cbf-9f1d-194630e85870-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:19.729543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.729448 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktlx\" (UniqueName: \"kubernetes.io/projected/90ae684f-bc2b-4cbf-9f1d-194630e85870-kube-api-access-fktlx\") pod \"payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:19.793860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.793195 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:19.823547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.808579 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk"] Oct 30 01:02:19.823547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.819708 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ae684f-bc2b-4cbf-9f1d-194630e85870-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:19.837167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.836307 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:19.931823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.924714 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktlx\" (UniqueName: \"kubernetes.io/projected/90ae684f-bc2b-4cbf-9f1d-194630e85870-kube-api-access-fktlx\") pod \"payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:19.931823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.929053 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk"] Oct 30 01:02:19.982810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.982757 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1951e859-93e1-4b7e-bf19-94cd6f25cbee-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:19.983415 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:19.983354 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xd7\" (UniqueName: \"kubernetes.io/projected/1951e859-93e1-4b7e-bf19-94cd6f25cbee-kube-api-access-s2xd7\") pod \"payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:20.122545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.110068 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xd7\" (UniqueName: \"kubernetes.io/projected/1951e859-93e1-4b7e-bf19-94cd6f25cbee-kube-api-access-s2xd7\") pod \"payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:20.122545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.110265 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1951e859-93e1-4b7e-bf19-94cd6f25cbee-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:20.124590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.124312 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:20.126301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.126116 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1951e859-93e1-4b7e-bf19-94cd6f25cbee-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:20.175209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.173209 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xd7\" (UniqueName: \"kubernetes.io/projected/1951e859-93e1-4b7e-bf19-94cd6f25cbee-kube-api-access-s2xd7\") pod \"payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:20.275268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.275225 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" event={"ID":"a33cb223-2191-4064-87d5-0ce005175455","Type":"ContainerStarted","Data":"27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2"} Oct 30 01:02:20.486111 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.486078 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:20.490255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.490234 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw"] Oct 30 01:02:20.492165 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:02:20.492132 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acc83b2_74c4_4499_a96b_3eecead8223d.slice/crio-b8da34cd8eb5eb9f3d38dc1dd45ccc268cf405329a02e8021389e80748a2cfda WatchSource:0}: Error finding container b8da34cd8eb5eb9f3d38dc1dd45ccc268cf405329a02e8021389e80748a2cfda: Status 404 returned error can't find the container with id b8da34cd8eb5eb9f3d38dc1dd45ccc268cf405329a02e8021389e80748a2cfda Oct 30 01:02:20.500105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.500068 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:20.500217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.500161 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:20.500217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.500210 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:20.588998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.588934 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8"] Oct 30 01:02:20.664418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.662736 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:02:20.664608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.664572 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:02:20.665279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.664626 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:02:20.873140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:20.873111 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk"] Oct 30 01:02:21.301472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.301432 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" event={"ID":"1951e859-93e1-4b7e-bf19-94cd6f25cbee","Type":"ContainerStarted","Data":"3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac"} Oct 30 01:02:21.301983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.301493 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" event={"ID":"1951e859-93e1-4b7e-bf19-94cd6f25cbee","Type":"ContainerStarted","Data":"58ba1dae65d7dd73ce1e57860c355e37d1b01acf2707f5818fef8f0a420d14a5"} Oct 30 01:02:21.301983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.301653 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:21.304702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.304667 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" event={"ID":"4acc83b2-74c4-4499-a96b-3eecead8223d","Type":"ContainerStarted","Data":"c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6"} Oct 30 01:02:21.304841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.304727 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" event={"ID":"4acc83b2-74c4-4499-a96b-3eecead8223d","Type":"ContainerStarted","Data":"b8da34cd8eb5eb9f3d38dc1dd45ccc268cf405329a02e8021389e80748a2cfda"} Oct 30 01:02:21.306575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.306507 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" event={"ID":"90ae684f-bc2b-4cbf-9f1d-194630e85870","Type":"ContainerStarted","Data":"70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887"} Oct 30 01:02:21.306703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.306596 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" event={"ID":"90ae684f-bc2b-4cbf-9f1d-194630e85870","Type":"ContainerStarted","Data":"fed1f00f24d5a578dba925cc177a041c62f41e8d4695aa2d723eac01c8dc2f54"} Oct 30 01:02:21.306766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.306747 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:21.386241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.386181 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" podStartSLOduration=2.386159133 podStartE2EDuration="2.386159133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:21.363847706 +0000 UTC m=+1426.307020898" watchObservedRunningTime="2026-10-30 01:02:21.386159133 +0000 UTC m=+1426.329332326" Oct 30 01:02:21.423487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.423428 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podStartSLOduration=2.423408798 podStartE2EDuration="2.423408798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:21.416811194 +0000 UTC m=+1426.359984388" watchObservedRunningTime="2026-10-30 01:02:21.423408798 +0000 UTC m=+1426.366581990" Oct 30 01:02:21.475580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.465563 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" podStartSLOduration=2.465544961 podStartE2EDuration="2.465544961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:21.463922782 +0000 UTC m=+1426.407095975" watchObservedRunningTime="2026-10-30 01:02:21.465544961 +0000 UTC m=+1426.408718151" Oct 30 01:02:21.639009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.638920 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:02:21.640591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.640557 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:02:21.810358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.810312 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:21.822557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.816908 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:02:21.822557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:21.817057 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:02:22.053636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:22.053604 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm"] Oct 30 01:02:22.140552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:22.139648 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f07412-djzzm"] Oct 30 01:02:22.632557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:22.632081 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:02:22.632557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:22.632137 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:02:22.795962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:22.795922 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:02:22.796830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:22.796793 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:02:23.196587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:23.196548 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e96bfd-f543-4590-85bc-1c6a5597c82e" path="/var/lib/kubelet/pods/b5e96bfd-f543-4590-85bc-1c6a5597c82e/volumes" Oct 30 01:02:23.637545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:23.635847 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:02:23.637545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:23.635919 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:02:23.796411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:23.795853 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:02:23.796411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:23.795908 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:02:24.638546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:24.628127 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:02:24.638546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:24.628225 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:02:24.798297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:24.798264 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:02:24.799574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:24.799536 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:02:25.390550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:25.390493 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs"] Oct 30 01:02:25.409631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:25.409597 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v"] Oct 30 01:02:25.521578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:25.520489 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:02:25.635698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:25.635661 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:02:25.656066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:25.653719 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:02:25.801539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:25.801484 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:02:25.803723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:25.803682 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:02:26.640544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.626137 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:02:26.640544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.626195 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:02:26.794545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.793771 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:26.794545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.794028 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 01:02:26.834764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.795142 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:02:26.834764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.795188 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:02:26.920760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.920649 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:26.921333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:26.921301 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:02:27.105543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:27.084276 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.139956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:27.139915 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.254247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:27.246725 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.690171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:27.690087 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:02:27.708204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:27.708170 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:02:27.796546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:27.796483 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:02:27.796891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:27.796576 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:02:28.801122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:28.801089 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:28.801771 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:28.801744 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:28.802458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:28.802437 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:02:30.032051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:30.032005 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj"] Oct 30 01:02:30.042318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:30.042283 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7507093-8dhgj"] Oct 30 01:02:30.390919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:30.390786 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" podUID="d06939e6-3c32-4a14-9e73-579e77c04808" containerName="payload-processing" containerID="cri-o://d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd" gracePeriod=25 Oct 30 01:02:30.409882 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:30.409824 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" podUID="985442ab-ef94-4ab5-8301-720ba216d8b4" containerName="payload-pre-processing" containerID="cri-o://dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2" gracePeriod=25 Oct 30 01:02:31.031611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:31.031577 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65"] Oct 30 01:02:31.044285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:31.044254 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22a2165-qxm65"] Oct 30 01:02:31.195325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:31.195291 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e8e981-01ed-4171-95f5-13be50c61f86" path="/var/lib/kubelet/pods/84e8e981-01ed-4171-95f5-13be50c61f86/volumes" Oct 30 01:02:31.196012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:31.195992 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9803fb11-307e-412b-b066-3cc75c834bd3" path="/var/lib/kubelet/pods/9803fb11-307e-412b-b066-3cc75c834bd3/volumes" Oct 30 01:02:32.322454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:32.322426 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:02:32.322952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:32.322495 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:02:32.735945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:32.735907 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" podUID="985442ab-ef94-4ab5-8301-720ba216d8b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:33.750385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:33.750340 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" podUID="d06939e6-3c32-4a14-9e73-579e77c04808" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:40.307457 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.307424 4905 scope.go:122] "RemoveContainer" containerID="35463962c0f930aab1da36b26f19575f8dfc535e2884c0a4ad58ecaa8dfc6cd6" Oct 30 01:02:40.319153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.319131 4905 scope.go:122] "RemoveContainer" containerID="81a2f128f84492863eb695e489183b41f586497b592c08fb9b1bac6a5aff29b2" Oct 30 01:02:40.339128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.339101 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq"] Oct 30 01:02:40.341105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.341085 4905 scope.go:122] "RemoveContainer" containerID="3b303c46d8e28ae206e0e6ecf512c0d03fe11fe4087de38ca4d242d7dd9d4506" Oct 30 01:02:40.344073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.344053 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.358155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.358137 4905 scope.go:122] "RemoveContainer" containerID="4766157e0bbbebf8866f78695525bdd057d86cadefe621b0225350f57d3122df" Oct 30 01:02:40.363643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.363619 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq"] Oct 30 01:02:40.366452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.364222 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:02:40.366452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.364343 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:02:40.366452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.364435 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:02:40.372890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.372043 4905 scope.go:122] "RemoveContainer" containerID="e13c4b0c400c1b5dcf7c3adf69d958b0f6c8ed1e134cc935f4ae2551fb9fbe70" Oct 30 01:02:40.374380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.374207 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:02:40.692605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.692489 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/d7561b2a-41d8-4238-8627-0eb82e85794c-kube-api-access-2rsfn\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.693133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.692995 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.693623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.693589 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.749730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.749424 4905 scope.go:122] "RemoveContainer" containerID="019577037a79d8fc607f6ec9a2eee53514ae8650d36f025622862cdee310a325" Oct 30 01:02:40.764790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.764767 4905 scope.go:122] "RemoveContainer" containerID="d549673f49083d97c4718c0f9902a49495484fa90506d5f70375da43d61bd004" Oct 30 01:02:40.776590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.776563 4905 scope.go:122] "RemoveContainer" containerID="b052c851b2f0eea470d02be7d687c454cab9ef5129a8551456f079f626aecdcf" Oct 30 01:02:40.790094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.790075 4905 scope.go:122] "RemoveContainer" containerID="48c3e2a3cd06f631f58c7b716c38110651795b18338115ef61c3b024379f57d7" Oct 30 01:02:40.795269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.795238 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.795429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.795412 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/d7561b2a-41d8-4238-8627-0eb82e85794c-kube-api-access-2rsfn\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.795589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.795573 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.796354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.796333 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.799025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.799001 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.801716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.801689 4905 scope.go:122] "RemoveContainer" containerID="e9f8c8d6bac77313686ad9bb8787b9cb28f395e2751633ab92fb97014e2a2878" Oct 30 01:02:40.830184 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.817149 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/d7561b2a-41d8-4238-8627-0eb82e85794c-kube-api-access-2rsfn\") pod \"maas-api-revoke-sub-d1d19f7f-jrhwq\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:40.830184 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.822397 4905 scope.go:122] "RemoveContainer" containerID="d168272ec28d9bdbfc6ff36c35493cbfb6abb7ca67c245f69b8c8cf558e9e624" Oct 30 01:02:40.863830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.851655 4905 scope.go:122] "RemoveContainer" containerID="44ad240404b4a84ae272f5b381417a4f812083189a858b4a2d1775e770950ffa" Oct 30 01:02:40.876716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.876689 4905 scope.go:122] "RemoveContainer" containerID="db724e2c9bbe62f30825e2e8c501b628d38358ae2cc23bfb44eaa9408b21e094" Oct 30 01:02:40.889092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.889072 4905 scope.go:122] "RemoveContainer" containerID="9264f35003b9a122b9c55f2544ece478d975fcb79ede8d8d6bb5f5840d99a990" Oct 30 01:02:40.965512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:40.965485 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:41.969009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:41.968978 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq"] Oct 30 01:02:42.424321 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:02:42.424280 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:02:42.780544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:42.774039 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" podUID="985442ab-ef94-4ab5-8301-720ba216d8b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:43.186544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.183788 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm"] Oct 30 01:02:43.369544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.368959 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" podStartSLOduration=3.368939209 podStartE2EDuration="3.368939209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:43.368710623 +0000 UTC m=+1448.311883816" watchObservedRunningTime="2026-10-30 01:02:43.368939209 +0000 UTC m=+1448.312112401" Oct 30 01:02:43.412546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.371264 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" event={"ID":"d7561b2a-41d8-4238-8627-0eb82e85794c","Type":"ContainerStarted","Data":"1924fc15f195197c810eaf7ad58957087e26b5e8ef98a0128726da9e9c56b4d9"} Oct 30 01:02:43.413720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.413690 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9db09f1-2rhrm"] Oct 30 01:02:43.413928 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.413909 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" event={"ID":"d7561b2a-41d8-4238-8627-0eb82e85794c","Type":"ContainerStarted","Data":"ad7b722c293294f41481cb1f5a134062cb0c4edc706f6e5c0e1fc52781f8fef3"} Oct 30 01:02:43.791601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.788205 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" podUID="d06939e6-3c32-4a14-9e73-579e77c04808" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:43.884558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.874624 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:43.884558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.874716 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 01:02:43.902548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:43.900272 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:44.096595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.095612 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:02:44.161467 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.161413 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5"] Oct 30 01:02:44.239602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.239558 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b7a97b-nfnb5"] Oct 30 01:02:44.328538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.328475 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:44.379018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.328773 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:44.567545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.538694 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:44.750603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.736100 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:44.752097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.752065 4905 generic.go:356] "Generic (PLEG): container finished" podID="d7561b2a-41d8-4238-8627-0eb82e85794c" containerID="1924fc15f195197c810eaf7ad58957087e26b5e8ef98a0128726da9e9c56b4d9" exitCode=0 Oct 30 01:02:44.752736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.752712 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" event={"ID":"d7561b2a-41d8-4238-8627-0eb82e85794c","Type":"ContainerDied","Data":"1924fc15f195197c810eaf7ad58957087e26b5e8ef98a0128726da9e9c56b4d9"} Oct 30 01:02:44.799827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.799725 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:02:44.907571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.900713 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:44.907571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:44.901368 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:02:45.188599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.179616 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:02:45.188599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.180432 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:02:45.220782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.220733 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:02:45.297584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.296999 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:02:45.543558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.540616 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:45.575551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.575462 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:45.598543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.596603 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:45.721929 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.721888 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f154eb95-5060-435d-8d11-bee2f83b2f10" path="/var/lib/kubelet/pods/f154eb95-5060-435d-8d11-bee2f83b2f10/volumes" Oct 30 01:02:45.744233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.723041 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f734b749-d3b3-41ff-a277-28721bafc52a" path="/var/lib/kubelet/pods/f734b749-d3b3-41ff-a277-28721bafc52a/volumes" Oct 30 01:02:45.771385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.771354 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:02:45.771676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.771658 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:02:45.771853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.771839 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:02:45.772028 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.772001 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:45.782562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.775755 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:02:45.783319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.783296 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:02:45.903312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.903213 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 01:02:45.924859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.924821 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:02:45.929543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.928072 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:02:45.953924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:45.933968 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:02:46.234598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.189133 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:46.292992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.249068 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:02:46.293602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.293576 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:02:46.359545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.343691 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:46.380598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.380020 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:02:46.561895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.561797 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:46.587790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.583437 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:46.660704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.609458 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:02:46.793255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.793205 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:46.802269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.799591 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:02:46.802269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.801212 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:46.964768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:46.964663 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:02:47.018552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.013623 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 01:02:47.296708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.271983 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:02:47.300672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.300553 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:02:47.384683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.384645 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:47.401962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.394310 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:02:47.625546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.572484 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:47.628105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.576707 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:02:47.629019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.591944 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:02:47.630141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.615992 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:02:47.666795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.663919 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:47.840773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.837235 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:47.840773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.838304 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-xks-ca\") pod \"d7561b2a-41d8-4238-8627-0eb82e85794c\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " Oct 30 01:02:47.840773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.838410 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7561b2a-41d8-4238-8627-0eb82e85794c-kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/d7561b2a-41d8-4238-8627-0eb82e85794c-kube-api-access-2rsfn\") pod \"d7561b2a-41d8-4238-8627-0eb82e85794c\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " Oct 30 01:02:47.840773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.838547 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-openshift-ca\") pod \"d7561b2a-41d8-4238-8627-0eb82e85794c\" (UID: \"d7561b2a-41d8-4238-8627-0eb82e85794c\") " Oct 30 01:02:47.853543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.850198 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:02:47.882542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.878928 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-openshift-ca" pod "d7561b2a-41d8-4238-8627-0eb82e85794c" (UID: "d7561b2a-41d8-4238-8627-0eb82e85794c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:47.893602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.893123 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:02:47.901549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.900262 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:02:47.902701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.902659 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7561b2a-41d8-4238-8627-0eb82e85794c-kube-api-access-2rsfn" pod "d7561b2a-41d8-4238-8627-0eb82e85794c" (UID: "d7561b2a-41d8-4238-8627-0eb82e85794c"). InnerVolumeSpecName "kube-api-access-2rsfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:47.928785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.928760 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:02:47.951548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.945866 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/d7561b2a-41d8-4238-8627-0eb82e85794c-kube-api-access-2rsfn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:47.951548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.945898 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:47.995963 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.995038 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:02:47.998126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:47.998084 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-xks-ca" pod "d7561b2a-41d8-4238-8627-0eb82e85794c" (UID: "d7561b2a-41d8-4238-8627-0eb82e85794c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:48.003046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.001991 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:48.054556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.021043 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:02:48.054556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.022050 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:02:48.208851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.185852 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7561b2a-41d8-4238-8627-0eb82e85794c-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:48.208851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.208109 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:02:48.293825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.293791 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" event={"ID":"d7561b2a-41d8-4238-8627-0eb82e85794c","Type":"ContainerDied","Data":"ad7b722c293294f41481cb1f5a134062cb0c4edc706f6e5c0e1fc52781f8fef3"} Oct 30 01:02:48.294276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.294245 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad7b722c293294f41481cb1f5a134062cb0c4edc706f6e5c0e1fc52781f8fef3" Oct 30 01:02:48.295768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.295741 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq" Oct 30 01:02:48.296588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.296571 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:02:48.322714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.315006 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:02:48.411558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.410312 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4"] Oct 30 01:02:48.418032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:48.417239 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94c1de-v52x4"] Oct 30 01:02:49.194665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.194633 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f582bd9e-7b3b-4d70-b614-03671b5eb034" path="/var/lib/kubelet/pods/f582bd9e-7b3b-4d70-b614-03671b5eb034/volumes" Oct 30 01:02:49.473429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.473343 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx"] Oct 30 01:02:49.474539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.474493 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d7561b2a-41d8-4238-8627-0eb82e85794c" containerName="revoke-keys" Oct 30 01:02:49.474539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.474537 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7561b2a-41d8-4238-8627-0eb82e85794c" containerName="revoke-keys" Oct 30 01:02:49.474741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.474720 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7561b2a-41d8-4238-8627-0eb82e85794c" containerName="revoke-keys" Oct 30 01:02:49.479725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.478269 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.495368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.495312 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx"] Oct 30 01:02:49.499895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.498759 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-43d152-openshift-default-dockercfg-5w798\"" Oct 30 01:02:49.575169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.575129 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.575477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.575457 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-token\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.575643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.575629 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-data\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.575749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.575735 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61f6648-27e9-448f-8e07-e2ed4341a36a-istiod-ca-cert\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.575918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.575904 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-certs\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.576092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.576080 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-credential-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.576292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.576277 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbqz\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-kube-api-access-sjbqz\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.576390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.576378 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-podinfo\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.576696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.576667 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-envoy\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.695879 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696012 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-token\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696055 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-data\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696085 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61f6648-27e9-448f-8e07-e2ed4341a36a-istiod-ca-cert\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696184 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-certs\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696289 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-credential-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696414 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbqz\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-kube-api-access-sjbqz\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696485 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-podinfo\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.697545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.696591 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-envoy\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.734591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.722942 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-certs\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.734591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.723565 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61f6648-27e9-448f-8e07-e2ed4341a36a-istiod-ca-cert\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.734591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.723802 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.734591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.724031 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-credential-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.734591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.732849 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-data\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.762541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.752121 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-podinfo\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.803543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.798429 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-envoy\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.803543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.803067 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-token\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:49.860558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:49.850044 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbqz\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-kube-api-access-sjbqz\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:50.125609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:50.120853 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:51.123932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.104818 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:51.125429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.125368 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:51.127091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.127061 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:51.130499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.110320 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx"] Oct 30 01:02:51.167334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.167306 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:02:51.172413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.172392 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:02:51.197857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.197827 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06939e6-3c32-4a14-9e73-579e77c04808-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06939e6-3c32-4a14-9e73-579e77c04808-plugins-config-volume\") pod \"d06939e6-3c32-4a14-9e73-579e77c04808\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " Oct 30 01:02:51.199060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.198058 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985442ab-ef94-4ab5-8301-720ba216d8b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985442ab-ef94-4ab5-8301-720ba216d8b4-plugins-config-volume\") pod \"985442ab-ef94-4ab5-8301-720ba216d8b4\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " Oct 30 01:02:51.199060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.198133 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06939e6-3c32-4a14-9e73-579e77c04808-plugins-config-volume" pod "d06939e6-3c32-4a14-9e73-579e77c04808" (UID: "d06939e6-3c32-4a14-9e73-579e77c04808"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:51.199060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.198162 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985442ab-ef94-4ab5-8301-720ba216d8b4-kube-api-access-kbm2q\" (UniqueName: \"kubernetes.io/projected/985442ab-ef94-4ab5-8301-720ba216d8b4-kube-api-access-kbm2q\") pod \"985442ab-ef94-4ab5-8301-720ba216d8b4\" (UID: \"985442ab-ef94-4ab5-8301-720ba216d8b4\") " Oct 30 01:02:51.199060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.198259 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06939e6-3c32-4a14-9e73-579e77c04808-kube-api-access-xthp2\" (UniqueName: \"kubernetes.io/projected/d06939e6-3c32-4a14-9e73-579e77c04808-kube-api-access-xthp2\") pod \"d06939e6-3c32-4a14-9e73-579e77c04808\" (UID: \"d06939e6-3c32-4a14-9e73-579e77c04808\") " Oct 30 01:02:51.199060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.198461 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985442ab-ef94-4ab5-8301-720ba216d8b4-plugins-config-volume" pod "985442ab-ef94-4ab5-8301-720ba216d8b4" (UID: "985442ab-ef94-4ab5-8301-720ba216d8b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:51.199060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.198996 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06939e6-3c32-4a14-9e73-579e77c04808-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:51.199060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.199021 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985442ab-ef94-4ab5-8301-720ba216d8b4-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:51.202845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.202014 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985442ab-ef94-4ab5-8301-720ba216d8b4-kube-api-access-kbm2q" pod "985442ab-ef94-4ab5-8301-720ba216d8b4" (UID: "985442ab-ef94-4ab5-8301-720ba216d8b4"). InnerVolumeSpecName "kube-api-access-kbm2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:51.204279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.204254 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06939e6-3c32-4a14-9e73-579e77c04808-kube-api-access-xthp2" pod "d06939e6-3c32-4a14-9e73-579e77c04808" (UID: "d06939e6-3c32-4a14-9e73-579e77c04808"). InnerVolumeSpecName "kube-api-access-xthp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:51.301107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.301066 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbm2q\" (UniqueName: \"kubernetes.io/projected/985442ab-ef94-4ab5-8301-720ba216d8b4-kube-api-access-kbm2q\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:51.301353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.301334 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xthp2\" (UniqueName: \"kubernetes.io/projected/d06939e6-3c32-4a14-9e73-579e77c04808-kube-api-access-xthp2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:02:51.475115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.473128 4905 generic.go:356] "Generic (PLEG): container finished" podID="985442ab-ef94-4ab5-8301-720ba216d8b4" containerID="dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2" exitCode=0 Oct 30 01:02:51.475115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.473329 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" event={"ID":"985442ab-ef94-4ab5-8301-720ba216d8b4","Type":"ContainerDied","Data":"dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2"} Oct 30 01:02:51.475115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.473381 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" event={"ID":"985442ab-ef94-4ab5-8301-720ba216d8b4","Type":"ContainerDied","Data":"901ca2527e426334c17dc9f61f6ba3bbef08f7eaa235a3adf569b00d29070a36"} Oct 30 01:02:51.475115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.473410 4905 scope.go:122] "RemoveContainer" containerID="dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2" Oct 30 01:02:51.475115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.473598 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v" Oct 30 01:02:51.478322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.478297 4905 generic.go:356] "Generic (PLEG): container finished" podID="d06939e6-3c32-4a14-9e73-579e77c04808" containerID="d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd" exitCode=0 Oct 30 01:02:51.478583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.478420 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" Oct 30 01:02:51.478846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.478440 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" event={"ID":"d06939e6-3c32-4a14-9e73-579e77c04808","Type":"ContainerDied","Data":"d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd"} Oct 30 01:02:51.478985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.478970 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs" event={"ID":"d06939e6-3c32-4a14-9e73-579e77c04808","Type":"ContainerDied","Data":"486e15ec81fda8961d14a4562ff162893867153121b2d4b859e88b29dbeda45d"} Oct 30 01:02:51.490339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.489352 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" event={"ID":"e61f6648-27e9-448f-8e07-e2ed4341a36a","Type":"ContainerStarted","Data":"9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9"} Oct 30 01:02:51.490339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.489404 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" event={"ID":"e61f6648-27e9-448f-8e07-e2ed4341a36a","Type":"ContainerStarted","Data":"7d1254732cc51cfeed83194203471743fd3f88f91fcd1a7c7aa0b03cc1348ee4"} Oct 30 01:02:51.494953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.494757 4905 scope.go:122] "RemoveContainer" containerID="dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2" Oct 30 01:02:51.495445 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:02:51.495417 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2\": container with ID starting with dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2 not found: ID does not exist" containerID="dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2" Oct 30 01:02:51.495646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.495454 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2"} err="failed to get container status \"dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2\": rpc error: code = NotFound desc = could not find container \"dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2\": container with ID starting with dc3a580d7f5477df10d6d16cb33e0e7b3b49186f3a88d9bd28127eb7ed6313a2 not found: ID does not exist" Oct 30 01:02:51.495646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.495476 4905 scope.go:122] "RemoveContainer" containerID="d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd" Oct 30 01:02:51.512017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.511947 4905 scope.go:122] "RemoveContainer" containerID="d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd" Oct 30 01:02:51.512375 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:02:51.512337 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd\": container with ID starting with d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd not found: ID does not exist" containerID="d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd" Oct 30 01:02:51.512463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.512384 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd"} err="failed to get container status \"d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd\": rpc error: code = NotFound desc = could not find container \"d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd\": container with ID starting with d668354e46f7f44b8b41c58728690d5a5c9b910af8f7650176be628c597833cd not found: ID does not exist" Oct 30 01:02:51.543079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.543013 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs"] Oct 30 01:02:51.552064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.552012 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-2zvcs"] Oct 30 01:02:51.589540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.588852 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podStartSLOduration=2.588834302 podStartE2EDuration="2.588834302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:51.583719392 +0000 UTC m=+1456.526892585" watchObservedRunningTime="2026-10-30 01:02:51.588834302 +0000 UTC m=+1456.532007499" Oct 30 01:02:51.631749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.630680 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v"] Oct 30 01:02:51.639440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:51.639408 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-5xt7v"] Oct 30 01:02:52.121796 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:52.121764 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:52.129142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:52.129114 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:52.503400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:52.503364 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:52.504443 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:52.504422 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:02:53.195761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:53.195726 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985442ab-ef94-4ab5-8301-720ba216d8b4" path="/var/lib/kubelet/pods/985442ab-ef94-4ab5-8301-720ba216d8b4/volumes" Oct 30 01:02:53.196410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:53.196353 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06939e6-3c32-4a14-9e73-579e77c04808" path="/var/lib/kubelet/pods/d06939e6-3c32-4a14-9e73-579e77c04808/volumes" Oct 30 01:02:53.421659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:53.421625 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:53.421830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:53.421704 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:55.893447 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.893363 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:02:55.893447 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.893423 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:02:55.905084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.905057 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj"] Oct 30 01:02:55.906051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.906031 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d06939e6-3c32-4a14-9e73-579e77c04808" containerName="payload-processing" Oct 30 01:02:55.906129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.906054 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06939e6-3c32-4a14-9e73-579e77c04808" containerName="payload-processing" Oct 30 01:02:55.906129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.906087 4905 cpu_manager.go:414] "containerMap: removing container" podUID="985442ab-ef94-4ab5-8301-720ba216d8b4" containerName="payload-pre-processing" Oct 30 01:02:55.906129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.906095 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985442ab-ef94-4ab5-8301-720ba216d8b4" containerName="payload-pre-processing" Oct 30 01:02:55.906304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.906293 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06939e6-3c32-4a14-9e73-579e77c04808" containerName="payload-processing" Oct 30 01:02:55.906351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.906313 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="985442ab-ef94-4ab5-8301-720ba216d8b4" containerName="payload-pre-processing" Oct 30 01:02:55.909095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.909079 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:55.925730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.925705 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-43d152\"" Oct 30 01:02:55.930509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.930475 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7nd9\" (UniqueName: \"kubernetes.io/projected/32177166-9202-4892-973a-99115094adc9-kube-api-access-j7nd9\") pod \"payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:55.930780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.930751 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32177166-9202-4892-973a-99115094adc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:55.932733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.932710 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj"] Oct 30 01:02:55.939873 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:55.939851 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-43d152-dockercfg-xzng9\"" Oct 30 01:02:56.117542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.117005 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32177166-9202-4892-973a-99115094adc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:56.117542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.117357 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7nd9\" (UniqueName: \"kubernetes.io/projected/32177166-9202-4892-973a-99115094adc9-kube-api-access-j7nd9\") pod \"payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:56.130539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.118508 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32177166-9202-4892-973a-99115094adc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:56.265148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.260334 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7nd9\" (UniqueName: \"kubernetes.io/projected/32177166-9202-4892-973a-99115094adc9-kube-api-access-j7nd9\") pod \"payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:56.339765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.339732 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm"] Oct 30 01:02:56.347197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.347172 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.354157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.354119 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm"] Oct 30 01:02:56.439583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.432808 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f444w\" (UniqueName: \"kubernetes.io/projected/db932632-d98d-4b76-9be3-318e2be88e3a-kube-api-access-f444w\") pod \"payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.439583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.432920 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db932632-d98d-4b76-9be3-318e2be88e3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.523702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.523608 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:56.535350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.535319 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f444w\" (UniqueName: \"kubernetes.io/projected/db932632-d98d-4b76-9be3-318e2be88e3a-kube-api-access-f444w\") pod \"payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.535713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.535677 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db932632-d98d-4b76-9be3-318e2be88e3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.536542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.536489 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db932632-d98d-4b76-9be3-318e2be88e3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.557096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.557065 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f444w\" (UniqueName: \"kubernetes.io/projected/db932632-d98d-4b76-9be3-318e2be88e3a-kube-api-access-f444w\") pod \"payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.598897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.598868 4905 generic.go:356] "Generic (PLEG): container finished" podID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerID="f090cbb7561550b044e6daf15c65c84321903c3b4086737276882711a88e8a3c" exitCode=0 Oct 30 01:02:56.599072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.599034 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" event={"ID":"0cc90b0e-4f0c-4104-8860-849aaf6772dc","Type":"ContainerDied","Data":"f090cbb7561550b044e6daf15c65c84321903c3b4086737276882711a88e8a3c"} Oct 30 01:02:56.599495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.599476 4905 scope.go:122] "RemoveContainer" containerID="f090cbb7561550b044e6daf15c65c84321903c3b4086737276882711a88e8a3c" Oct 30 01:02:56.602767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.602257 4905 generic.go:356] "Generic (PLEG): container finished" podID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerID="b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680" exitCode=0 Oct 30 01:02:56.602767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.602417 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" event={"ID":"4bcdfb72-abd3-4976-930f-b96e5887ccef","Type":"ContainerDied","Data":"b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680"} Oct 30 01:02:56.603542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.603428 4905 scope.go:122] "RemoveContainer" containerID="b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680" Oct 30 01:02:56.604160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.603953 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:56.604160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.604039 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:56.604160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.604086 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:56.607561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.606802 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:56.607561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.606877 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:56.607561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.606918 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:02:56.785741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.785655 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:56.865060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:56.865018 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj"] Oct 30 01:02:57.210314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.210219 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm"] Oct 30 01:02:57.622644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.622360 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" event={"ID":"4bcdfb72-abd3-4976-930f-b96e5887ccef","Type":"ContainerStarted","Data":"f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a"} Oct 30 01:02:57.638875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.638830 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" event={"ID":"32177166-9202-4892-973a-99115094adc9","Type":"ContainerStarted","Data":"a935c0805bc64d424ad0303b7cdc8785d2b4df3524ce580fb2f0113e2b4cd407"} Oct 30 01:02:57.639148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.639124 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" event={"ID":"32177166-9202-4892-973a-99115094adc9","Type":"ContainerStarted","Data":"3a8275c0f0dcb229bdcad71f931fc7dd3b5a88ea235fe531e1f666d90198a4de"} Oct 30 01:02:57.640925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.640885 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:02:57.658574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.658538 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" event={"ID":"0cc90b0e-4f0c-4104-8860-849aaf6772dc","Type":"ContainerStarted","Data":"b9b5ae3cacba9a885898a340b60b04e020287a8a267615087d7815b42cc06422"} Oct 30 01:02:57.673137 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.673099 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" event={"ID":"db932632-d98d-4b76-9be3-318e2be88e3a","Type":"ContainerStarted","Data":"69675fefb7a7421f38687c218ac74d5505c071edf1ab8724e77a3f64315f6b33"} Oct 30 01:02:57.673410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.673386 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" event={"ID":"db932632-d98d-4b76-9be3-318e2be88e3a","Type":"ContainerStarted","Data":"5916a3296d5388bdd42bbd2783dd94395708727007d5d3d7fd741aff043e43d7"} Oct 30 01:02:57.674644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.674620 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:02:57.722617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.722566 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" podStartSLOduration=2.7225526589999998 podStartE2EDuration="2.722552659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:57.720762636 +0000 UTC m=+1462.663935827" watchObservedRunningTime="2026-10-30 01:02:57.722552659 +0000 UTC m=+1462.665725845" Oct 30 01:02:57.749545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.749415 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:02:57.749767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.749578 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:02:57.749767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.749620 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:02:57.781628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:57.780890 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" podStartSLOduration=2.780875524 podStartE2EDuration="2.780875524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:57.775633756 +0000 UTC m=+1462.718806949" watchObservedRunningTime="2026-10-30 01:02:57.780875524 +0000 UTC m=+1462.724048744" Oct 30 01:02:58.124360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:58.124322 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:02:58.133696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:58.133661 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:02:58.133971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:58.133939 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 01:02:58.757545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:58.743845 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:02:58.757545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:58.743903 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:02:59.134591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:59.133861 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:02:59.134591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:59.133916 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 01:02:59.774897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:59.774858 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:02:59.797771 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:02:59.797730 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:03:00.132593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:00.126241 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:03:00.132593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:00.126312 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 01:03:00.747547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:00.744059 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:03:00.747547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:00.744123 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:03:01.132603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.128803 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:03:01.132603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.128860 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 01:03:01.493725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.493676 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq"] Oct 30 01:03:01.503097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.503070 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.508203 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.508106 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq"] Oct 30 01:03:01.588044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.588012 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:03:01.589076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.588690 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:03:01.589232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.588714 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:03:01.589321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.588922 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:03:01.617822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.617787 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqx8\" (UniqueName: \"kubernetes.io/projected/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-kube-api-access-tfqx8\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.618076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.618055 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.618189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.618120 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.720301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.720267 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.720513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.720332 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.720809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.720785 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqx8\" (UniqueName: \"kubernetes.io/projected/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-kube-api-access-tfqx8\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.721103 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.721075 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.723080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.723054 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.756629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.755697 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:03:01.756629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.755752 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:03:01.797575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.796378 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqx8\" (UniqueName: \"kubernetes.io/projected/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-kube-api-access-tfqx8\") pod \"maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:01.893544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:01.893179 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:02.130381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:02.125365 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:03:02.130381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:02.125425 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 01:03:02.781549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:02.776443 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:03:02.781549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:02.778764 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:03:02.781549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:02.781481 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq"] Oct 30 01:03:02.894031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:02.893997 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" event={"ID":"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e","Type":"ContainerStarted","Data":"34e87cb923cc8fd225da8dec184d2323ef522718ad18abbaef1bdd11c5f6023f"} Oct 30 01:03:03.147600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:03.146876 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:03:03.147600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:03.146932 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 01:03:03.745544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:03.743609 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:03:03.745544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:03.744500 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:03:03.745544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:03.745265 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:03:03.920807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:03.920636 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" event={"ID":"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e","Type":"ContainerStarted","Data":"f09d3b1892b44e0934c270366802b53927ca2351e7d9a21714acd65fac63627c"} Oct 30 01:03:04.009943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:03.981632 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" podStartSLOduration=2.981608134 podStartE2EDuration="2.981608134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:03.971838299 +0000 UTC m=+1468.915011491" watchObservedRunningTime="2026-10-30 01:03:03.981608134 +0000 UTC m=+1468.924781332" Oct 30 01:03:04.155798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:04.141293 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:03:04.155798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:04.142874 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:03:04.155798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:04.147097 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:03:04.931033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:04.930999 4905 generic.go:356] "Generic (PLEG): container finished" podID="ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" containerID="f09d3b1892b44e0934c270366802b53927ca2351e7d9a21714acd65fac63627c" exitCode=0 Oct 30 01:03:04.931535 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:04.931082 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" event={"ID":"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e","Type":"ContainerDied","Data":"f09d3b1892b44e0934c270366802b53927ca2351e7d9a21714acd65fac63627c"} Oct 30 01:03:06.198098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.198076 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:06.291981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.291896 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-xks-ca\") pod \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " Oct 30 01:03:06.292224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.292205 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-openshift-ca\") pod \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " Oct 30 01:03:06.292291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.292275 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-kube-api-access-tfqx8\" (UniqueName: \"kubernetes.io/projected/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-kube-api-access-tfqx8\") pod \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\" (UID: \"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e\") " Oct 30 01:03:06.292588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.292561 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-openshift-ca" pod "ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" (UID: "ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:06.293297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.293273 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:06.294828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.294795 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-xks-ca" pod "ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" (UID: "ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:03:06.295436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.295413 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-kube-api-access-tfqx8" pod "ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" (UID: "ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e"). InnerVolumeSpecName "kube-api-access-tfqx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:06.395032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.395003 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:06.395220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.395037 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqx8\" (UniqueName: \"kubernetes.io/projected/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e-kube-api-access-tfqx8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:06.982215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.982183 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" event={"ID":"ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e","Type":"ContainerDied","Data":"34e87cb923cc8fd225da8dec184d2323ef522718ad18abbaef1bdd11c5f6023f"} Oct 30 01:03:06.982215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.982218 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq" Oct 30 01:03:06.982457 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:06.982240 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e87cb923cc8fd225da8dec184d2323ef522718ad18abbaef1bdd11c5f6023f" Oct 30 01:03:07.032648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:07.032619 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz"] Oct 30 01:03:07.038061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:07.038032 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43d0110-xlcqz"] Oct 30 01:03:07.045047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:07.045021 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85"] Oct 30 01:03:07.049216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:07.049186 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9116d3ed-zth85"] Oct 30 01:03:07.192976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:07.192946 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64ea606-fc60-41d2-aa91-c7d8352e9988" path="/var/lib/kubelet/pods/c64ea606-fc60-41d2-aa91-c7d8352e9988/volumes" Oct 30 01:03:07.193482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:07.193467 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d761e306-1293-4a66-ab30-82658e2ab0fb" path="/var/lib/kubelet/pods/d761e306-1293-4a66-ab30-82658e2ab0fb/volumes" Oct 30 01:03:09.804145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.804117 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:03:09.804712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.804186 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:03:09.810425 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.810399 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4"] Oct 30 01:03:09.811204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.811187 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" containerName="revoke-keys" Oct 30 01:03:09.811251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.811206 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" containerName="revoke-keys" Oct 30 01:03:09.811383 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.811373 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" containerName="revoke-keys" Oct 30 01:03:09.816639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.816621 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.824660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.824630 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.825041 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.824909 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.825182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.825163 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdx8\" (UniqueName: \"kubernetes.io/projected/38797784-bbdc-42d1-9bbb-02baac6b96ac-kube-api-access-ckdx8\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.835652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.833759 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:03:09.850402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.850374 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4"] Oct 30 01:03:09.850552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.850508 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:03:09.850632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.850605 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:03:09.850732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.850715 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:03:09.927363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.927326 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdx8\" (UniqueName: \"kubernetes.io/projected/38797784-bbdc-42d1-9bbb-02baac6b96ac-kube-api-access-ckdx8\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.927577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.927496 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.927707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.927687 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.928508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.928478 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.930950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.930926 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:09.953571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:09.953531 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdx8\" (UniqueName: \"kubernetes.io/projected/38797784-bbdc-42d1-9bbb-02baac6b96ac-kube-api-access-ckdx8\") pod \"maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:10.162206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.162134 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:10.201896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.201858 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn"] Oct 30 01:03:10.209196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.209171 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.235555 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.235652 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-token\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.235759 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9v8\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-kube-api-access-df9v8\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.235828 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-podinfo\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.235904 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-credential-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.235971 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9370e3cd-a5c6-4b32-833b-68195297a32f-istiod-ca-cert\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.236098 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-certs\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.236135 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-envoy\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.236248 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-data\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.240351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.239276 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn"] Oct 30 01:03:10.254554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.253949 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-b05110-openshift-default-dockercfg-jlvbk\"" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.337876 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-certs\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.337969 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-envoy\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.338115 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-data\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.338193 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.338221 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-token\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.338328 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9v8\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-kube-api-access-df9v8\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.338420 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-podinfo\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.338462 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-credential-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.338560 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9370e3cd-a5c6-4b32-833b-68195297a32f-istiod-ca-cert\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.339591 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9370e3cd-a5c6-4b32-833b-68195297a32f-istiod-ca-cert\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.414593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.339867 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-certs\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.709550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.657191 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-envoy\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.709550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.673416 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-data\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.709550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.673880 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.709550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.680363 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-credential-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.887569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.885056 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-token\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.903544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.892429 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-podinfo\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.908740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.907967 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9v8\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-kube-api-access-df9v8\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:10.996787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:10.996752 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4"] Oct 30 01:03:11.045780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:11.045749 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" event={"ID":"38797784-bbdc-42d1-9bbb-02baac6b96ac","Type":"ContainerStarted","Data":"0893a97386d0a98631517080ab087367968b69d0c15c2589631f1c06c21520bf"} Oct 30 01:03:11.157045 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:11.157015 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:11.540328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:11.540292 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:11.540428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:11.540365 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:11.540428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:11.540399 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:11.544872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:11.544844 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn"] Oct 30 01:03:12.064742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.064642 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" event={"ID":"38797784-bbdc-42d1-9bbb-02baac6b96ac","Type":"ContainerStarted","Data":"cada77802c7375e81aecdd29fe4acca0f1991c77bbdae1bd99567fe1aa6fb404"} Oct 30 01:03:12.066766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.066737 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" event={"ID":"9370e3cd-a5c6-4b32-833b-68195297a32f","Type":"ContainerStarted","Data":"9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a"} Oct 30 01:03:12.066896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.066785 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" event={"ID":"9370e3cd-a5c6-4b32-833b-68195297a32f","Type":"ContainerStarted","Data":"d5cf4ebc1b913dc057398c49c162628a13ba5eabf8e8737bb4fd0d891acd44a6"} Oct 30 01:03:12.109426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.109376 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" podStartSLOduration=3.109362368 podStartE2EDuration="3.109362368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:12.10731711 +0000 UTC m=+1477.050490302" watchObservedRunningTime="2026-10-30 01:03:12.109362368 +0000 UTC m=+1477.052535560" Oct 30 01:03:12.133019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.132981 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx"] Oct 30 01:03:12.159225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.157863 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:12.165121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.165096 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:12.168826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:12.168777 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" podStartSLOduration=2.168764338 podStartE2EDuration="2.168764338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:12.156510195 +0000 UTC m=+1477.099683388" watchObservedRunningTime="2026-10-30 01:03:12.168764338 +0000 UTC m=+1477.111937529" Oct 30 01:03:12.549238 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:12.549201 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:03:13.238544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:13.237549 4905 generic.go:356] "Generic (PLEG): container finished" podID="38797784-bbdc-42d1-9bbb-02baac6b96ac" containerID="cada77802c7375e81aecdd29fe4acca0f1991c77bbdae1bd99567fe1aa6fb404" exitCode=0 Oct 30 01:03:13.248083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:13.248043 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" event={"ID":"38797784-bbdc-42d1-9bbb-02baac6b96ac","Type":"ContainerDied","Data":"cada77802c7375e81aecdd29fe4acca0f1991c77bbdae1bd99567fe1aa6fb404"} Oct 30 01:03:13.248252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:13.248104 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:13.248252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:13.248144 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:03:14.440734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.440705 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:14.465202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.465170 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-xks-ca\") pod \"38797784-bbdc-42d1-9bbb-02baac6b96ac\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " Oct 30 01:03:14.465357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.465251 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-openshift-ca\") pod \"38797784-bbdc-42d1-9bbb-02baac6b96ac\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " Oct 30 01:03:14.465501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.465487 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38797784-bbdc-42d1-9bbb-02baac6b96ac-kube-api-access-ckdx8\" (UniqueName: \"kubernetes.io/projected/38797784-bbdc-42d1-9bbb-02baac6b96ac-kube-api-access-ckdx8\") pod \"38797784-bbdc-42d1-9bbb-02baac6b96ac\" (UID: \"38797784-bbdc-42d1-9bbb-02baac6b96ac\") " Oct 30 01:03:14.465671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.465651 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-openshift-ca" pod "38797784-bbdc-42d1-9bbb-02baac6b96ac" (UID: "38797784-bbdc-42d1-9bbb-02baac6b96ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:14.466739 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.466712 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:14.467628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.467608 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-xks-ca" pod "38797784-bbdc-42d1-9bbb-02baac6b96ac" (UID: "38797784-bbdc-42d1-9bbb-02baac6b96ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:03:14.467827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.467809 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38797784-bbdc-42d1-9bbb-02baac6b96ac-kube-api-access-ckdx8" pod "38797784-bbdc-42d1-9bbb-02baac6b96ac" (UID: "38797784-bbdc-42d1-9bbb-02baac6b96ac"). InnerVolumeSpecName "kube-api-access-ckdx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:14.568501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.568421 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38797784-bbdc-42d1-9bbb-02baac6b96ac-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:14.568501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:14.568451 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdx8\" (UniqueName: \"kubernetes.io/projected/38797784-bbdc-42d1-9bbb-02baac6b96ac-kube-api-access-ckdx8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:15.365805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:15.365723 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" Oct 30 01:03:15.461466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:15.461434 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4" event={"ID":"38797784-bbdc-42d1-9bbb-02baac6b96ac","Type":"ContainerDied","Data":"0893a97386d0a98631517080ab087367968b69d0c15c2589631f1c06c21520bf"} Oct 30 01:03:15.461958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:15.461938 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0893a97386d0a98631517080ab087367968b69d0c15c2589631f1c06c21520bf" Oct 30 01:03:16.600053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.600014 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5"] Oct 30 01:03:16.604710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.601950 4905 cpu_manager.go:414] "containerMap: removing container" podUID="38797784-bbdc-42d1-9bbb-02baac6b96ac" containerName="revoke-keys" Oct 30 01:03:16.604710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.601988 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38797784-bbdc-42d1-9bbb-02baac6b96ac" containerName="revoke-keys" Oct 30 01:03:16.604710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.602271 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="38797784-bbdc-42d1-9bbb-02baac6b96ac" containerName="revoke-keys" Oct 30 01:03:16.661735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.661703 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5"] Oct 30 01:03:16.662058 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.662018 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:16.785721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.784598 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-b05110\"" Oct 30 01:03:16.785721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.785243 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-b05110-dockercfg-7swsr\"" Oct 30 01:03:16.967599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.962711 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jpf\" (UniqueName: \"kubernetes.io/projected/407de83a-8478-47d5-8fce-4ea9509263bb-kube-api-access-p8jpf\") pod \"payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:16.967599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:16.963063 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407de83a-8478-47d5-8fce-4ea9509263bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:17.109352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.101988 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407de83a-8478-47d5-8fce-4ea9509263bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:17.109352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.102254 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jpf\" (UniqueName: \"kubernetes.io/projected/407de83a-8478-47d5-8fce-4ea9509263bb-kube-api-access-p8jpf\") pod \"payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:17.109352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.103954 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407de83a-8478-47d5-8fce-4ea9509263bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:17.135155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.134843 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jpf\" (UniqueName: \"kubernetes.io/projected/407de83a-8478-47d5-8fce-4ea9509263bb-kube-api-access-p8jpf\") pod \"payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:17.135155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.134916 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" podUID="c8fbe00b-2b6a-4550-9e51-9be201cd82c5" containerName="payload-pre-processing" containerID="cri-o://60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b" gracePeriod=25 Oct 30 01:03:17.342788 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:03:17.342697 4905 logging.go:55] [core] [Channel #1526 SubChannel #1527]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.171:50052", ServerName: "10.132.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.171:50052: connect: connection reset by peer" Oct 30 01:03:17.343214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.343185 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6"] Oct 30 01:03:17.349024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.348993 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" containerID="cri-o://93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe" gracePeriod=30 Oct 30 01:03:17.428703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.428651 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:17.727670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.727645 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:03:17.781915 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:03:17.781880 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407de83a_8478_47d5_8fce_4ea9509263bb.slice/crio-014a3761cf6329df69171033abfce4e3fccd21f565a753e5960e568311e5a7e7 WatchSource:0}: Error finding container 014a3761cf6329df69171033abfce4e3fccd21f565a753e5960e568311e5a7e7: Status 404 returned error can't find the container with id 014a3761cf6329df69171033abfce4e3fccd21f565a753e5960e568311e5a7e7 Oct 30 01:03:17.784646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.784625 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5"] Oct 30 01:03:17.829906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.829877 4905 generic.go:356] "Generic (PLEG): container finished" podID="c8fbe00b-2b6a-4550-9e51-9be201cd82c5" containerID="60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b" exitCode=0 Oct 30 01:03:17.830033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.829927 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" Oct 30 01:03:17.830033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.829966 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" event={"ID":"c8fbe00b-2b6a-4550-9e51-9be201cd82c5","Type":"ContainerDied","Data":"60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b"} Oct 30 01:03:17.830033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.830025 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" event={"ID":"c8fbe00b-2b6a-4550-9e51-9be201cd82c5","Type":"ContainerDied","Data":"901f454bc11eeac927c163c247820b6aa79331a31369b4102bde5da9fdf97d78"} Oct 30 01:03:17.830191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.830052 4905 scope.go:122] "RemoveContainer" containerID="60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b" Oct 30 01:03:17.831816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.831759 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" event={"ID":"407de83a-8478-47d5-8fce-4ea9509263bb","Type":"ContainerStarted","Data":"014a3761cf6329df69171033abfce4e3fccd21f565a753e5960e568311e5a7e7"} Oct 30 01:03:17.849228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.849205 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-kube-api-access-d4574\" (UniqueName: \"kubernetes.io/projected/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-kube-api-access-d4574\") pod \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " Oct 30 01:03:17.849427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.849410 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-plugins-config-volume\") pod \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\" (UID: \"c8fbe00b-2b6a-4550-9e51-9be201cd82c5\") " Oct 30 01:03:17.849804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.849771 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-plugins-config-volume" pod "c8fbe00b-2b6a-4550-9e51-9be201cd82c5" (UID: "c8fbe00b-2b6a-4550-9e51-9be201cd82c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:17.850361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.850330 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:17.851750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.851727 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-kube-api-access-d4574" pod "c8fbe00b-2b6a-4550-9e51-9be201cd82c5" (UID: "c8fbe00b-2b6a-4550-9e51-9be201cd82c5"). InnerVolumeSpecName "kube-api-access-d4574". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:17.853083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.852961 4905 scope.go:122] "RemoveContainer" containerID="60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b" Oct 30 01:03:17.853381 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:17.853353 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b\": container with ID starting with 60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b not found: ID does not exist" containerID="60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b" Oct 30 01:03:17.853501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.853391 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b"} err="failed to get container status \"60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b\": rpc error: code = NotFound desc = could not find container \"60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b\": container with ID starting with 60674c3ce7c8e52ce8adbd9782a09568e2f04bb6a6d848d5153d8c2eb5c76d4b not found: ID does not exist" Oct 30 01:03:17.960569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.956177 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4574\" (UniqueName: \"kubernetes.io/projected/c8fbe00b-2b6a-4550-9e51-9be201cd82c5-kube-api-access-d4574\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:17.964138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:17.963801 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx" podUID="c8fbe00b-2b6a-4550-9e51-9be201cd82c5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.171:50052\" within 1s: context deadline exceeded" Oct 30 01:03:18.194980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:18.194951 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx"] Oct 30 01:03:18.200860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:18.200834 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7452d0-6dc94fd7f5-w76xx"] Oct 30 01:03:18.842192 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:18.842154 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" event={"ID":"407de83a-8478-47d5-8fce-4ea9509263bb","Type":"ContainerStarted","Data":"378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a"} Oct 30 01:03:18.842647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:18.842217 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:18.884723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:18.884675 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" podStartSLOduration=2.884662525 podStartE2EDuration="2.884662525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:18.88214322 +0000 UTC m=+1483.825316410" watchObservedRunningTime="2026-10-30 01:03:18.884662525 +0000 UTC m=+1483.827835716" Oct 30 01:03:19.196674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:19.196582 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fbe00b-2b6a-4550-9e51-9be201cd82c5" path="/var/lib/kubelet/pods/c8fbe00b-2b6a-4550-9e51-9be201cd82c5/volumes" Oct 30 01:03:20.492544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:20.489554 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs"] Oct 30 01:03:20.510701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:20.510671 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-rmpxs"] Oct 30 01:03:21.116565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:21.116493 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4"] Oct 30 01:03:21.153048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:21.153012 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d"] Oct 30 01:03:21.195407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:21.195379 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b4ed5f-cc87-42c3-b252-9a20545c8a4e" path="/var/lib/kubelet/pods/72b4ed5f-cc87-42c3-b252-9a20545c8a4e/volumes" Oct 30 01:03:23.032716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.032653 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl"] Oct 30 01:03:23.042727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.041385 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0404966b-dz8gl"] Oct 30 01:03:23.087403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.087372 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189000 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-envoy\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189141 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43af3985-3f75-48bb-92a6-b025bdc61f8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43af3985-3f75-48bb-92a6-b025bdc61f8a-istiod-ca-cert\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189189 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-token\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-token\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189234 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-certs\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189272 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-credential-socket\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189320 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-data\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189347 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-podinfo\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189410 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-kube-api-access-qdn74\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-kube-api-access-qdn74\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.189601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.189469 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-socket\") pod \"43af3985-3f75-48bb-92a6-b025bdc61f8a\" (UID: \"43af3985-3f75-48bb-92a6-b025bdc61f8a\") " Oct 30 01:03:23.196580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.190432 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-socket" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:23.196580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.192332 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-credential-socket" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:23.196580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.196340 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-data" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:23.198729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.192720 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43af3985-3f75-48bb-92a6-b025bdc61f8a-istiod-ca-cert" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:23.200634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.200227 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-envoy" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:23.200634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.200261 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-certs" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:23.203550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.202647 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-kube-api-access-qdn74" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "kube-api-access-qdn74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:23.203550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.203002 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-podinfo" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:23.204095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.204050 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-token" pod "43af3985-3f75-48bb-92a6-b025bdc61f8a" (UID: "43af3985-3f75-48bb-92a6-b025bdc61f8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:23.210979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.209991 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb02dfe-1464-4f18-9333-b867c4ecd34b" path="/var/lib/kubelet/pods/9cb02dfe-1464-4f18-9333-b867c4ecd34b/volumes" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.366994 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367047 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43af3985-3f75-48bb-92a6-b025bdc61f8a-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367060 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367073 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367085 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367097 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367110 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43af3985-3f75-48bb-92a6-b025bdc61f8a-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367122 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdn74\" (UniqueName: \"kubernetes.io/projected/43af3985-3f75-48bb-92a6-b025bdc61f8a-kube-api-access-qdn74\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.367562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.367139 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43af3985-3f75-48bb-92a6-b025bdc61f8a-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:23.503574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.503482 4905 generic.go:356] "Generic (PLEG): container finished" podID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerID="93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe" exitCode=0 Oct 30 01:03:23.504657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.504632 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" Oct 30 01:03:23.504818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.504787 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" event={"ID":"43af3985-3f75-48bb-92a6-b025bdc61f8a","Type":"ContainerDied","Data":"93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe"} Oct 30 01:03:23.504883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.504845 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6" event={"ID":"43af3985-3f75-48bb-92a6-b025bdc61f8a","Type":"ContainerDied","Data":"69826887693fad294a83768bd321883c48fa127ba3377121fdc7d4ff5f3c6e94"} Oct 30 01:03:23.504934 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.504896 4905 scope.go:122] "RemoveContainer" containerID="93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe" Oct 30 01:03:23.517367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.517351 4905 scope.go:122] "RemoveContainer" containerID="93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe" Oct 30 01:03:23.517691 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:23.517669 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe\": container with ID starting with 93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe not found: ID does not exist" containerID="93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe" Oct 30 01:03:23.517787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.517700 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe"} err="failed to get container status \"93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe\": rpc error: code = NotFound desc = could not find container \"93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe\": container with ID starting with 93169f64eae7e677bd807e2eade4a4083ca8cd00043da1f22429ae95f2729ffe not found: ID does not exist" Oct 30 01:03:23.577666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.577631 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6"] Oct 30 01:03:23.589910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:23.589876 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-6zlb6"] Oct 30 01:03:25.194935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.194905 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" path="/var/lib/kubelet/pods/43af3985-3f75-48bb-92a6-b025bdc61f8a/volumes" Oct 30 01:03:25.321755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.321715 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km"] Oct 30 01:03:25.323360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.323334 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c8fbe00b-2b6a-4550-9e51-9be201cd82c5" containerName="payload-pre-processing" Oct 30 01:03:25.323360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.323360 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fbe00b-2b6a-4550-9e51-9be201cd82c5" containerName="payload-pre-processing" Oct 30 01:03:25.323605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.323379 4905 cpu_manager.go:414] "containerMap: removing container" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" Oct 30 01:03:25.323605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.323389 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" Oct 30 01:03:25.323736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.323660 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fbe00b-2b6a-4550-9e51-9be201cd82c5" containerName="payload-pre-processing" Oct 30 01:03:25.323736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.323676 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="43af3985-3f75-48bb-92a6-b025bdc61f8a" containerName="istio-proxy" Oct 30 01:03:25.389853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.389757 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km"] Oct 30 01:03:25.390040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.389949 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.407580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.407554 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:03:25.408026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.408004 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:03:25.408345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.408315 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:03:25.421132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.421101 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:03:25.526572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.526536 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.526907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.526883 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4qn\" (UniqueName: \"kubernetes.io/projected/fdc88044-2936-45ab-a3a1-9bb59e3267a8-kube-api-access-2k4qn\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.540931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.540894 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.666607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.666566 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.667760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.667735 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4qn\" (UniqueName: \"kubernetes.io/projected/fdc88044-2936-45ab-a3a1-9bb59e3267a8-kube-api-access-2k4qn\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.691768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.691731 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.694575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.667625 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.710599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.709228 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4qn\" (UniqueName: \"kubernetes.io/projected/fdc88044-2936-45ab-a3a1-9bb59e3267a8-kube-api-access-2k4qn\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.715305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.715199 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b814fb-gw4km\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:25.715542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:25.715506 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:26.116468 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:03:26.116431 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc88044_2936_45ab_a3a1_9bb59e3267a8.slice/crio-3b2c65c13cfe4816a4171c97618093f9aee2f84f346186c57b0ceb6d6a5f4a84 WatchSource:0}: Error finding container 3b2c65c13cfe4816a4171c97618093f9aee2f84f346186c57b0ceb6d6a5f4a84: Status 404 returned error can't find the container with id 3b2c65c13cfe4816a4171c97618093f9aee2f84f346186c57b0ceb6d6a5f4a84 Oct 30 01:03:26.117265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.117234 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" podUID="e8af2efd-365f-4419-8f25-8e7276bedd83" containerName="payload-processing" containerID="cri-o://932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39" gracePeriod=25 Oct 30 01:03:26.144266 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.121543 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km"] Oct 30 01:03:26.161539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.155628 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" podUID="0974e7dc-5e06-4e10-bf33-cfc27a12ea96" containerName="payload-pre-processing" containerID="cri-o://c7e65a8980b37b0286056e975421af3ec0aa4b0d7a1c2449a1bdb7c18a4c67db" gracePeriod=25 Oct 30 01:03:26.665754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.665724 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:03:26.690122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.690089 4905 generic.go:356] "Generic (PLEG): container finished" podID="0974e7dc-5e06-4e10-bf33-cfc27a12ea96" containerID="c7e65a8980b37b0286056e975421af3ec0aa4b0d7a1c2449a1bdb7c18a4c67db" exitCode=0 Oct 30 01:03:26.690347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.690320 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" event={"ID":"0974e7dc-5e06-4e10-bf33-cfc27a12ea96","Type":"ContainerDied","Data":"c7e65a8980b37b0286056e975421af3ec0aa4b0d7a1c2449a1bdb7c18a4c67db"} Oct 30 01:03:26.691832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.691807 4905 generic.go:356] "Generic (PLEG): container finished" podID="e8af2efd-365f-4419-8f25-8e7276bedd83" containerID="932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39" exitCode=0 Oct 30 01:03:26.691954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.691936 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" event={"ID":"e8af2efd-365f-4419-8f25-8e7276bedd83","Type":"ContainerDied","Data":"932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39"} Oct 30 01:03:26.692023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.691981 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" event={"ID":"e8af2efd-365f-4419-8f25-8e7276bedd83","Type":"ContainerDied","Data":"2fe4eb26d5c824ded80d51f1603b4f82c6f68362ada54a276d6bcb9973fb1e7b"} Oct 30 01:03:26.692023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.692013 4905 scope.go:122] "RemoveContainer" containerID="932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39" Oct 30 01:03:26.692327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.692174 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4" Oct 30 01:03:26.695671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.694786 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" event={"ID":"fdc88044-2936-45ab-a3a1-9bb59e3267a8","Type":"ContainerStarted","Data":"3b2c65c13cfe4816a4171c97618093f9aee2f84f346186c57b0ceb6d6a5f4a84"} Oct 30 01:03:26.755626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.736452 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8af2efd-365f-4419-8f25-8e7276bedd83-kube-api-access-vhwdb\" (UniqueName: \"kubernetes.io/projected/e8af2efd-365f-4419-8f25-8e7276bedd83-kube-api-access-vhwdb\") pod \"e8af2efd-365f-4419-8f25-8e7276bedd83\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " Oct 30 01:03:26.755626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.752870 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8af2efd-365f-4419-8f25-8e7276bedd83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af2efd-365f-4419-8f25-8e7276bedd83-plugins-config-volume\") pod \"e8af2efd-365f-4419-8f25-8e7276bedd83\" (UID: \"e8af2efd-365f-4419-8f25-8e7276bedd83\") " Oct 30 01:03:26.771539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.770814 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8af2efd-365f-4419-8f25-8e7276bedd83-plugins-config-volume" pod "e8af2efd-365f-4419-8f25-8e7276bedd83" (UID: "e8af2efd-365f-4419-8f25-8e7276bedd83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:26.771539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.771364 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8af2efd-365f-4419-8f25-8e7276bedd83-kube-api-access-vhwdb" pod "e8af2efd-365f-4419-8f25-8e7276bedd83" (UID: "e8af2efd-365f-4419-8f25-8e7276bedd83"). InnerVolumeSpecName "kube-api-access-vhwdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:26.856083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.855700 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af2efd-365f-4419-8f25-8e7276bedd83-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:26.856083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.855730 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwdb\" (UniqueName: \"kubernetes.io/projected/e8af2efd-365f-4419-8f25-8e7276bedd83-kube-api-access-vhwdb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:26.880666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.863127 4905 scope.go:122] "RemoveContainer" containerID="932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39" Oct 30 01:03:26.958754 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:26.941369 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39\": container with ID starting with 932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39 not found: ID does not exist" containerID="932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39" Oct 30 01:03:26.958754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:26.941418 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39"} err="failed to get container status \"932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39\": rpc error: code = NotFound desc = could not find container \"932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39\": container with ID starting with 932b5c8f2cf1cbeaafa1a455f8457fc61ee352e19fafe00401eda35ba70d4f39 not found: ID does not exist" Oct 30 01:03:27.413721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.413688 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:03:27.463561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.454669 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-plugins-config-volume\") pod \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " Oct 30 01:03:27.463561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.454866 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-kube-api-access-v7mnq\" (UniqueName: \"kubernetes.io/projected/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-kube-api-access-v7mnq\") pod \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\" (UID: \"0974e7dc-5e06-4e10-bf33-cfc27a12ea96\") " Oct 30 01:03:27.463561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.457455 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-plugins-config-volume" pod "0974e7dc-5e06-4e10-bf33-cfc27a12ea96" (UID: "0974e7dc-5e06-4e10-bf33-cfc27a12ea96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:27.463561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.461736 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4"] Oct 30 01:03:27.472258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.472198 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-kube-api-access-v7mnq" pod "0974e7dc-5e06-4e10-bf33-cfc27a12ea96" (UID: "0974e7dc-5e06-4e10-bf33-cfc27a12ea96"). InnerVolumeSpecName "kube-api-access-v7mnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:27.473657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.473628 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fcc0d3e0-856b6f995b-gvnq4"] Oct 30 01:03:27.560599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.557231 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:27.560599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.557266 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mnq\" (UniqueName: \"kubernetes.io/projected/0974e7dc-5e06-4e10-bf33-cfc27a12ea96-kube-api-access-v7mnq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:27.710026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.709999 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" Oct 30 01:03:27.710411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.710028 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d" event={"ID":"0974e7dc-5e06-4e10-bf33-cfc27a12ea96","Type":"ContainerDied","Data":"086f06ab2f966a5a29466009500b78828ef01b7e9004f36dffe04c15eccef0df"} Oct 30 01:03:27.710411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.710104 4905 scope.go:122] "RemoveContainer" containerID="c7e65a8980b37b0286056e975421af3ec0aa4b0d7a1c2449a1bdb7c18a4c67db" Oct 30 01:03:27.714193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.714171 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" event={"ID":"fdc88044-2936-45ab-a3a1-9bb59e3267a8","Type":"ContainerStarted","Data":"59e1568a0a0826eef12877c67cc7813425d5a6fbe2358c6b0c9e1675b0b1b456"} Oct 30 01:03:27.800948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.800903 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d"] Oct 30 01:03:27.814302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.814240 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fcc0d3e0-7584c6dcf5-7278d"] Oct 30 01:03:27.847730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:27.847196 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" podStartSLOduration=2.847178301 podStartE2EDuration="2.847178301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:27.83982428 +0000 UTC m=+1492.782997474" watchObservedRunningTime="2026-10-30 01:03:27.847178301 +0000 UTC m=+1492.790351494" Oct 30 01:03:28.729789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:28.729755 4905 generic.go:356] "Generic (PLEG): container finished" podID="fdc88044-2936-45ab-a3a1-9bb59e3267a8" containerID="59e1568a0a0826eef12877c67cc7813425d5a6fbe2358c6b0c9e1675b0b1b456" exitCode=0 Oct 30 01:03:28.730214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:28.729815 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" event={"ID":"fdc88044-2936-45ab-a3a1-9bb59e3267a8","Type":"ContainerDied","Data":"59e1568a0a0826eef12877c67cc7813425d5a6fbe2358c6b0c9e1675b0b1b456"} Oct 30 01:03:29.104154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.104123 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9"] Oct 30 01:03:29.104961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.104947 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e8af2efd-365f-4419-8f25-8e7276bedd83" containerName="payload-processing" Oct 30 01:03:29.105006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.104963 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8af2efd-365f-4419-8f25-8e7276bedd83" containerName="payload-processing" Oct 30 01:03:29.105063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.105017 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0974e7dc-5e06-4e10-bf33-cfc27a12ea96" containerName="payload-pre-processing" Oct 30 01:03:29.105063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.105023 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0974e7dc-5e06-4e10-bf33-cfc27a12ea96" containerName="payload-pre-processing" Oct 30 01:03:29.105165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.105155 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8af2efd-365f-4419-8f25-8e7276bedd83" containerName="payload-processing" Oct 30 01:03:29.105602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.105179 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0974e7dc-5e06-4e10-bf33-cfc27a12ea96" containerName="payload-pre-processing" Oct 30 01:03:29.109688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.109661 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:29.125702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.125677 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-869c367e\"" Oct 30 01:03:29.129640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.129615 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9"] Oct 30 01:03:29.204552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.202401 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0974e7dc-5e06-4e10-bf33-cfc27a12ea96" path="/var/lib/kubelet/pods/0974e7dc-5e06-4e10-bf33-cfc27a12ea96/volumes" Oct 30 01:03:29.204552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.203046 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8af2efd-365f-4419-8f25-8e7276bedd83" path="/var/lib/kubelet/pods/e8af2efd-365f-4419-8f25-8e7276bedd83/volumes" Oct 30 01:03:29.204552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.203506 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67"] Oct 30 01:03:29.226545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.210274 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67"] Oct 30 01:03:29.226545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.210382 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:29.226545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.223618 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15671937-b5d5-4ae5-9aef-53f1a3892717-plugins-config-volume\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:29.226545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.223730 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hws4l\" (UniqueName: \"kubernetes.io/projected/15671937-b5d5-4ae5-9aef-53f1a3892717-kube-api-access-hws4l\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:29.404600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.403924 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-869c367e-dockercfg-42dgr\"" Oct 30 01:03:29.435542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.431771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skklr\" (UniqueName: \"kubernetes.io/projected/250d7d06-3e88-4efb-8190-07e6a776ddac-kube-api-access-skklr\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:29.435542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.431838 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250d7d06-3e88-4efb-8190-07e6a776ddac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:29.435542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.431915 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15671937-b5d5-4ae5-9aef-53f1a3892717-plugins-config-volume\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:29.435542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.432010 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hws4l\" (UniqueName: \"kubernetes.io/projected/15671937-b5d5-4ae5-9aef-53f1a3892717-kube-api-access-hws4l\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:29.435542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.433445 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15671937-b5d5-4ae5-9aef-53f1a3892717-plugins-config-volume\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:29.557544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.533907 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skklr\" (UniqueName: \"kubernetes.io/projected/250d7d06-3e88-4efb-8190-07e6a776ddac-kube-api-access-skklr\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:29.728599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.661394 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250d7d06-3e88-4efb-8190-07e6a776ddac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:29.728599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.662946 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250d7d06-3e88-4efb-8190-07e6a776ddac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:29.728599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.703474 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skklr\" (UniqueName: \"kubernetes.io/projected/250d7d06-3e88-4efb-8190-07e6a776ddac-kube-api-access-skklr\") pod \"payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:29.740593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.736877 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hws4l\" (UniqueName: \"kubernetes.io/projected/15671937-b5d5-4ae5-9aef-53f1a3892717-kube-api-access-hws4l\") pod \"payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:29.807041 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:29.807017 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:30.040842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.035283 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:30.075554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.075493 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:30.177698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.177662 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-openshift-ca\") pod \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " Oct 30 01:03:30.178273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.178254 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc88044-2936-45ab-a3a1-9bb59e3267a8-kube-api-access-2k4qn\" (UniqueName: \"kubernetes.io/projected/fdc88044-2936-45ab-a3a1-9bb59e3267a8-kube-api-access-2k4qn\") pod \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " Oct 30 01:03:30.179055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.178263 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-openshift-ca" pod "fdc88044-2936-45ab-a3a1-9bb59e3267a8" (UID: "fdc88044-2936-45ab-a3a1-9bb59e3267a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:30.179213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.179193 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-xks-ca\") pod \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\" (UID: \"fdc88044-2936-45ab-a3a1-9bb59e3267a8\") " Oct 30 01:03:30.180322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.180298 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:30.183452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.180505 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67"] Oct 30 01:03:30.183452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.181606 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc88044-2936-45ab-a3a1-9bb59e3267a8-kube-api-access-2k4qn" pod "fdc88044-2936-45ab-a3a1-9bb59e3267a8" (UID: "fdc88044-2936-45ab-a3a1-9bb59e3267a8"). InnerVolumeSpecName "kube-api-access-2k4qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:30.184440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.184391 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-xks-ca" pod "fdc88044-2936-45ab-a3a1-9bb59e3267a8" (UID: "fdc88044-2936-45ab-a3a1-9bb59e3267a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:03:30.285857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.285816 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k4qn\" (UniqueName: \"kubernetes.io/projected/fdc88044-2936-45ab-a3a1-9bb59e3267a8-kube-api-access-2k4qn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:30.286135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.286110 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc88044-2936-45ab-a3a1-9bb59e3267a8-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:30.290270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.289054 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:03:30.481019 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:03:30.480972 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15671937_b5d5_4ae5_9aef_53f1a3892717.slice/crio-f49b0722e13b719d238f437a8e296f6bcfa9ce3669981d23c943fc9947c34127 WatchSource:0}: Error finding container f49b0722e13b719d238f437a8e296f6bcfa9ce3669981d23c943fc9947c34127: Status 404 returned error can't find the container with id f49b0722e13b719d238f437a8e296f6bcfa9ce3669981d23c943fc9947c34127 Oct 30 01:03:30.512798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.511461 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9"] Oct 30 01:03:30.804118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.804055 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" event={"ID":"250d7d06-3e88-4efb-8190-07e6a776ddac","Type":"ContainerStarted","Data":"07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67"} Oct 30 01:03:30.804634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.804359 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:30.804634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.804406 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" event={"ID":"250d7d06-3e88-4efb-8190-07e6a776ddac","Type":"ContainerStarted","Data":"a88500a7bf69c75900223af9a523e465e0f297afdf4c52d8cc57cbb0b6dcedae"} Oct 30 01:03:30.806187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.806156 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" event={"ID":"15671937-b5d5-4ae5-9aef-53f1a3892717","Type":"ContainerStarted","Data":"5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573"} Oct 30 01:03:30.806365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.806196 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" event={"ID":"15671937-b5d5-4ae5-9aef-53f1a3892717","Type":"ContainerStarted","Data":"f49b0722e13b719d238f437a8e296f6bcfa9ce3669981d23c943fc9947c34127"} Oct 30 01:03:30.806875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.806857 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:30.808375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.808352 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" event={"ID":"fdc88044-2936-45ab-a3a1-9bb59e3267a8","Type":"ContainerDied","Data":"3b2c65c13cfe4816a4171c97618093f9aee2f84f346186c57b0ceb6d6a5f4a84"} Oct 30 01:03:30.808481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.808393 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2c65c13cfe4816a4171c97618093f9aee2f84f346186c57b0ceb6d6a5f4a84" Oct 30 01:03:30.808481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.808463 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km" Oct 30 01:03:30.852612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.850875 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" podStartSLOduration=1.850857871 podStartE2EDuration="1.850857871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:30.844231386 +0000 UTC m=+1495.787404582" watchObservedRunningTime="2026-10-30 01:03:30.850857871 +0000 UTC m=+1495.794031064" Oct 30 01:03:30.887114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:30.887063 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" podStartSLOduration=1.887047295 podStartE2EDuration="1.887047295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:30.883674096 +0000 UTC m=+1495.826847289" watchObservedRunningTime="2026-10-30 01:03:30.887047295 +0000 UTC m=+1495.830220488" Oct 30 01:03:32.164717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.164677 4905 patch_prober.go:29] interesting pod/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:03:32.165153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.164743 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:03:32.310492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.308499 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd"] Oct 30 01:03:32.310492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.308773 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" containerID="cri-o://1664e06f01c5c5f3275a7d7a7f67ae122889c5dca64e8f4442ad29c836e5b7c8" gracePeriod=30 Oct 30 01:03:32.370850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.363037 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz"] Oct 30 01:03:32.377879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.376649 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" containerID="cri-o://fa08e786d89b46ab1ef9ec92b87cf7a1aca1fff96678132bbd1ba4dece0a4210" gracePeriod=30 Oct 30 01:03:32.408541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.390721 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6"] Oct 30 01:03:32.408541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.390980 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" containerID="cri-o://3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b" gracePeriod=30 Oct 30 01:03:32.456681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.455128 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx"] Oct 30 01:03:32.456681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.455365 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" containerID="cri-o://e4e39fef92c046edfd69bf920a93b24e55e092a3eebebaa7685a1ea90b304df1" gracePeriod=30 Oct 30 01:03:32.520640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.520604 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns"] Oct 30 01:03:32.520859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.520833 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" containerID="cri-o://27782b6555faa15ee8fc3fc11661f4a46f5e309bf03c420ccfcb7c07b93ee8fc" gracePeriod=30 Oct 30 01:03:32.525480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.525451 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x"] Oct 30 01:03:32.525712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.525689 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" containerID="cri-o://59541bda9b9e3d0bd462ac76ed3b22a3f6a50fb4fd1bc537fa29ffdb470eb0dd" gracePeriod=30 Oct 30 01:03:32.548007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.543344 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz"] Oct 30 01:03:32.548007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.543585 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" containerID="cri-o://d2fcd78cc64d7106d9603657fb66163aabd6c179a97144bf049db278b1b5dfdc" gracePeriod=30 Oct 30 01:03:32.551719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.551689 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r"] Oct 30 01:03:32.551904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:32.551881 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" containerID="cri-o://2446d6b5e1ce21c5a20a13f4eecdf575b8f556d28cc7f5fc8046dc278f446641" gracePeriod=30 Oct 30 01:03:34.547541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:34.527770 4905 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:03:34.547541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:34.528121 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:03:35.665767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.665734 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:03:35.668010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.667989 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:03:35.791855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.791820 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m"] Oct 30 01:03:35.938846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.938766 4905 generic.go:356] "Generic (PLEG): container finished" podID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerID="39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0" exitCode=0 Oct 30 01:03:35.939030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.938839 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" event={"ID":"5cd40be8-c62e-4831-8fe8-075ae7835375","Type":"ContainerDied","Data":"39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0"} Oct 30 01:03:35.939355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.939340 4905 scope.go:122] "RemoveContainer" containerID="39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0" Oct 30 01:03:35.941269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.941236 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:35.941387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.941319 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:35.941387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:35.941364 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:37.017544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:37.015592 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" event={"ID":"5cd40be8-c62e-4831-8fe8-075ae7835375","Type":"ContainerStarted","Data":"4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d"} Oct 30 01:03:37.338188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:37.259712 4905 patch_prober.go:29] interesting pod/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:03:37.338188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:37.259857 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:03:37.400731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:37.400686 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:03:38.090856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.089431 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:03:38.090856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.089485 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 01:03:38.102305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.099177 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 01:03:38.125904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.124327 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/948f189d-119b-4657-ab78-81373fffd7e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948f189d-119b-4657-ab78-81373fffd7e3-istio-podinfo\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.125904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.124476 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-socket\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.125904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.124574 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-credential-socket\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.125904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.124653 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-data\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.132642 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-socket" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.146683 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948f189d-119b-4657-ab78-81373fffd7e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948f189d-119b-4657-ab78-81373fffd7e3-istiod-ca-cert\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.146884 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-envoy\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.146977 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-istio-token\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-istio-token\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.147030 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-credential-socket" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.147083 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-certs\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.147148 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-kube-api-access-zhcl9\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-kube-api-access-zhcl9\") pod \"948f189d-119b-4657-ab78-81373fffd7e3\" (UID: \"948f189d-119b-4657-ab78-81373fffd7e3\") " Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.148097 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.148123 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.150669 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-certs" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.155763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.152395 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948f189d-119b-4657-ab78-81373fffd7e3-istiod-ca-cert" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.203395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.186372 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-data" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.203395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.194671 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-envoy" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.203395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.199857 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-istio-token" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.210079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.210049 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/948f189d-119b-4657-ab78-81373fffd7e3-istio-podinfo" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.223806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.223765 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-kube-api-access-zhcl9" pod "948f189d-119b-4657-ab78-81373fffd7e3" (UID: "948f189d-119b-4657-ab78-81373fffd7e3"). InnerVolumeSpecName "kube-api-access-zhcl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.232544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.232357 4905 patch_prober.go:29] interesting pod/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:03:38.232544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.232408 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:03:38.256551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.255589 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948f189d-119b-4657-ab78-81373fffd7e3-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.256551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.255621 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.256551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.255636 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948f189d-119b-4657-ab78-81373fffd7e3-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.256551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.255650 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.256551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.255662 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.256551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.255674 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948f189d-119b-4657-ab78-81373fffd7e3-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.256551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.255687 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcl9\" (UniqueName: \"kubernetes.io/projected/948f189d-119b-4657-ab78-81373fffd7e3-kube-api-access-zhcl9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.284441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.279803 4905 generic.go:356] "Generic (PLEG): container finished" podID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerID="59541bda9b9e3d0bd462ac76ed3b22a3f6a50fb4fd1bc537fa29ffdb470eb0dd" exitCode=0 Oct 30 01:03:38.284441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.279847 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" event={"ID":"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1","Type":"ContainerDied","Data":"59541bda9b9e3d0bd462ac76ed3b22a3f6a50fb4fd1bc537fa29ffdb470eb0dd"} Oct 30 01:03:38.284441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.279927 4905 scope.go:122] "RemoveContainer" containerID="9db5876a683516c0129d0c3344125381824bca37d5cb868c9bebf078b52107be" Oct 30 01:03:38.286246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.286219 4905 generic.go:356] "Generic (PLEG): container finished" podID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerID="fa08e786d89b46ab1ef9ec92b87cf7a1aca1fff96678132bbd1ba4dece0a4210" exitCode=0 Oct 30 01:03:38.286772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.286402 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" event={"ID":"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7","Type":"ContainerDied","Data":"fa08e786d89b46ab1ef9ec92b87cf7a1aca1fff96678132bbd1ba4dece0a4210"} Oct 30 01:03:38.314411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.314234 4905 generic.go:356] "Generic (PLEG): container finished" podID="948f189d-119b-4657-ab78-81373fffd7e3" containerID="3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b" exitCode=0 Oct 30 01:03:38.314776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.314389 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" Oct 30 01:03:38.316564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.314540 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" event={"ID":"948f189d-119b-4657-ab78-81373fffd7e3","Type":"ContainerDied","Data":"3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b"} Oct 30 01:03:38.316757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.316735 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6" event={"ID":"948f189d-119b-4657-ab78-81373fffd7e3","Type":"ContainerDied","Data":"af8e93789f976e313fc0bdd6d638625ec1a52a8a62740f0f9c09e32cd2b0178a"} Oct 30 01:03:38.375069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.375034 4905 generic.go:356] "Generic (PLEG): container finished" podID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerID="2446d6b5e1ce21c5a20a13f4eecdf575b8f556d28cc7f5fc8046dc278f446641" exitCode=0 Oct 30 01:03:38.375230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.375179 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" event={"ID":"8445bf3a-7f07-4bc5-a092-3eef8b985999","Type":"ContainerDied","Data":"2446d6b5e1ce21c5a20a13f4eecdf575b8f556d28cc7f5fc8046dc278f446641"} Oct 30 01:03:38.377267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.377235 4905 generic.go:356] "Generic (PLEG): container finished" podID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerID="1664e06f01c5c5f3275a7d7a7f67ae122889c5dca64e8f4442ad29c836e5b7c8" exitCode=0 Oct 30 01:03:38.377405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.377270 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" event={"ID":"22737c92-facc-49a0-b4a2-4980aacd0c36","Type":"ContainerDied","Data":"1664e06f01c5c5f3275a7d7a7f67ae122889c5dca64e8f4442ad29c836e5b7c8"} Oct 30 01:03:38.378348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.378318 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 01:03:38.383822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.383789 4905 generic.go:356] "Generic (PLEG): container finished" podID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerID="27782b6555faa15ee8fc3fc11661f4a46f5e309bf03c420ccfcb7c07b93ee8fc" exitCode=0 Oct 30 01:03:38.384317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.384292 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" event={"ID":"202a9b97-fc86-4e8d-b3e8-380942de9f32","Type":"ContainerDied","Data":"27782b6555faa15ee8fc3fc11661f4a46f5e309bf03c420ccfcb7c07b93ee8fc"} Oct 30 01:03:38.385902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.385885 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 01:03:38.400759 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.400735 4905 generic.go:356] "Generic (PLEG): container finished" podID="1106de83-4886-4b14-815a-6e8fb28d9621" containerID="d2fcd78cc64d7106d9603657fb66163aabd6c179a97144bf049db278b1b5dfdc" exitCode=0 Oct 30 01:03:38.401020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.401001 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" event={"ID":"1106de83-4886-4b14-815a-6e8fb28d9621","Type":"ContainerDied","Data":"d2fcd78cc64d7106d9603657fb66163aabd6c179a97144bf049db278b1b5dfdc"} Oct 30 01:03:38.403126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.403104 4905 generic.go:356] "Generic (PLEG): container finished" podID="804d523e-4a52-4960-9564-912a704258d2" containerID="e4e39fef92c046edfd69bf920a93b24e55e092a3eebebaa7685a1ea90b304df1" exitCode=0 Oct 30 01:03:38.404256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.404217 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" event={"ID":"804d523e-4a52-4960-9564-912a704258d2","Type":"ContainerDied","Data":"e4e39fef92c046edfd69bf920a93b24e55e092a3eebebaa7685a1ea90b304df1"} Oct 30 01:03:38.420246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.420071 4905 scope.go:122] "RemoveContainer" containerID="ff56d0289a87d165bbfcd464dd8c32411586c51736d0c07fe3395192f5787d85" Oct 30 01:03:38.573468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.573354 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 01:03:38.576133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.575921 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-token\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-token\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579254 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-socket\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579301 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istiod-ca-cert\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579339 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-kube-api-access-gg4b5\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-kube-api-access-gg4b5\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579380 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-credential-socket\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579417 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-token\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-token\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579460 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-socket\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579532 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-credential-socket\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.579564 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-credential-socket\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.580506 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istiod-ca-cert" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.580775 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-socket" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.581288 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-socket" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.584980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.583130 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-credential-socket" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.585259 4905 scope.go:122] "RemoveContainer" containerID="3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.585912 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-token\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-token\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.587546 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8445bf3a-7f07-4bc5-a092-3eef8b985999-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445bf3a-7f07-4bc5-a092-3eef8b985999-istiod-ca-cert\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.587620 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-podinfo\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.587919 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-credential-socket" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.588748 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-data\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.588790 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-data\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.588819 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-data\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.588871 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-kube-api-access-msrsm\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-kube-api-access-msrsm\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.588900 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22737c92-facc-49a0-b4a2-4980aacd0c36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22737c92-facc-49a0-b4a2-4980aacd0c36-istiod-ca-cert\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.588989 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-socket\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589031 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-certs\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589071 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-podinfo\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589128 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-envoy\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589156 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-certs\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589206 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-envoy\") pod \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\" (UID: \"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589234 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-kube-api-access-8zm5q\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-kube-api-access-8zm5q\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589276 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-podinfo\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589314 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-envoy\") pod \"22737c92-facc-49a0-b4a2-4980aacd0c36\" (UID: \"22737c92-facc-49a0-b4a2-4980aacd0c36\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589344 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-certs\") pod \"8445bf3a-7f07-4bc5-a092-3eef8b985999\" (UID: \"8445bf3a-7f07-4bc5-a092-3eef8b985999\") " Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.589818 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-credential-socket" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.590286 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-certs" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.590474 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-data" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.590701 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-data" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.590873 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-certs" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591675 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591697 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591711 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591725 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591739 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591753 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591766 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591779 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591791 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.591803 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.594510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.592024 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-data" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.606029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.605802 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-certs" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.607755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.607702 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8445bf3a-7f07-4bc5-a092-3eef8b985999-istiod-ca-cert" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.620731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.617887 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6"] Oct 30 01:03:38.620731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.618754 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-socket" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.620731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.619627 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-z2jp6"] Oct 30 01:03:38.621888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.621692 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22737c92-facc-49a0-b4a2-4980aacd0c36-istiod-ca-cert" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.668334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.656701 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-envoy" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.668334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.657325 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-envoy" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.674325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.673771 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-kube-api-access-8zm5q" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "kube-api-access-8zm5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.674325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.673850 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-podinfo" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.676613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.675030 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-token" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.676613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.675107 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-podinfo" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.676613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.675354 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-token" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.676613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.675421 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-kube-api-access-gg4b5" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "kube-api-access-gg4b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.677287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.677253 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-envoy" pod "8445bf3a-7f07-4bc5-a092-3eef8b985999" (UID: "8445bf3a-7f07-4bc5-a092-3eef8b985999"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.688544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.683642 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-podinfo" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.688544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.687491 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-kube-api-access-msrsm" pod "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" (UID: "d021244d-d6b8-4390-b1a1-5aafeaf6d0b1"). InnerVolumeSpecName "kube-api-access-msrsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.688544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.687671 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-token" pod "22737c92-facc-49a0-b4a2-4980aacd0c36" (UID: "22737c92-facc-49a0-b4a2-4980aacd0c36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701189 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701226 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701240 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701255 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701267 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701280 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zm5q\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-kube-api-access-8zm5q\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701294 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701306 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701318 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445bf3a-7f07-4bc5-a092-3eef8b985999-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701331 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4b5\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-kube-api-access-gg4b5\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701345 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701357 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701369 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445bf3a-7f07-4bc5-a092-3eef8b985999-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701380 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701392 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22737c92-facc-49a0-b4a2-4980aacd0c36-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.701407 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrsm\" (UniqueName: \"kubernetes.io/projected/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1-kube-api-access-msrsm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.702135 4905 scope.go:122] "RemoveContainer" containerID="3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:38.705292 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b\": container with ID starting with 3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b not found: ID does not exist" containerID="3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.705327 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b"} err="failed to get container status \"3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b\": rpc error: code = NotFound desc = could not find container \"3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b\": container with ID starting with 3e2f8ff681fb9e5e0b77073f3b4cf7645be417cabec0ebf2b113c7bb884dfa2b not found: ID does not exist" Oct 30 01:03:38.710259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.708030 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22737c92-facc-49a0-b4a2-4980aacd0c36-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.711911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.711791 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 01:03:38.718554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.717508 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 01:03:38.723180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.722491 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808567 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-credential-socket\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808618 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-podinfo\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808660 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-kube-api-access-vgbz8\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-kube-api-access-vgbz8\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808757 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-kube-api-access-twsfd\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-kube-api-access-twsfd\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808797 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-envoy\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808830 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-credential-socket" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808835 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-envoy\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808918 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-istio-token\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-istio-token\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.808951 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-certs\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809054 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-envoy\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809086 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-token\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-token\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809136 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1106de83-4886-4b14-815a-6e8fb28d9621-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106de83-4886-4b14-815a-6e8fb28d9621-istio-podinfo\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809201 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-token\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-token\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809232 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istiod-ca-cert\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809261 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-socket\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809306 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-podinfo\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809349 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-credential-socket\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809426 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-credential-socket\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809452 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202a9b97-fc86-4e8d-b3e8-380942de9f32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202a9b97-fc86-4e8d-b3e8-380942de9f32-istiod-ca-cert\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809501 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-socket\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809618 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-certs\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809877 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1106de83-4886-4b14-815a-6e8fb28d9621-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106de83-4886-4b14-815a-6e8fb28d9621-istiod-ca-cert\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.809948 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-data\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.810007 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-kube-api-access-wpf6r\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-kube-api-access-wpf6r\") pod \"202a9b97-fc86-4e8d-b3e8-380942de9f32\" (UID: \"202a9b97-fc86-4e8d-b3e8-380942de9f32\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.810039 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-data\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.810120 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-certs\") pod \"1106de83-4886-4b14-815a-6e8fb28d9621\" (UID: \"1106de83-4886-4b14-815a-6e8fb28d9621\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.810156 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-data\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.811145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.810189 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-socket\") pod \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\" (UID: \"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7\") " Oct 30 01:03:38.812441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.811811 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-envoy" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.812657 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.812693 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.813134 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-kube-api-access-vgbz8" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "kube-api-access-vgbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.813406 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-credential-socket" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.813656 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-certs" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.814067 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-certs" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.814289 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-data" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.814655 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1106de83-4886-4b14-815a-6e8fb28d9621-istiod-ca-cert" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.815273 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-credential-socket" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.815320 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-socket" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.815412 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istiod-ca-cert" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.815619 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-certs" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.817341 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-data" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.819071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.817495 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202a9b97-fc86-4e8d-b3e8-380942de9f32-istiod-ca-cert" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.820983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.819642 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-socket" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.820983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.820421 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-socket" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.820983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.820432 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-data" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.824053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.823989 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1106de83-4886-4b14-815a-6e8fb28d9621-istio-podinfo" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.824177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.824048 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-envoy" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.824177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.824086 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-kube-api-access-wpf6r" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "kube-api-access-wpf6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.824440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.824399 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-envoy" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.839627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.824492 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-istio-token" pod "1106de83-4886-4b14-815a-6e8fb28d9621" (UID: "1106de83-4886-4b14-815a-6e8fb28d9621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.839627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.832998 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-kube-api-access-twsfd" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "kube-api-access-twsfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.860817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.840991 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 01:03:38.861876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.861668 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-token" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.861876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.861780 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-token" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.861876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.861843 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-podinfo" pod "202a9b97-fc86-4e8d-b3e8-380942de9f32" (UID: "202a9b97-fc86-4e8d-b3e8-380942de9f32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.862613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.862573 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-podinfo" pod "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" (UID: "d3d11eb5-43ca-41a9-b945-3219c8e0bcc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.913532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.913493 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-credential-socket\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.913710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.913617 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-certs\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.913710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.913692 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-istio-token\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-istio-token\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.913830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.913765 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/804d523e-4a52-4960-9564-912a704258d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d523e-4a52-4960-9564-912a704258d2-istio-podinfo\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.913830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.913825 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-envoy\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.913934 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.913870 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-kube-api-access-cvrss\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-kube-api-access-cvrss\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.913984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.913943 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-socket\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.914041 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.914005 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804d523e-4a52-4960-9564-912a704258d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d523e-4a52-4960-9564-912a704258d2-istiod-ca-cert\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.914096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.914053 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-data\") pod \"804d523e-4a52-4960-9564-912a704258d2\" (UID: \"804d523e-4a52-4960-9564-912a704258d2\") " Oct 30 01:03:38.914978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.914351 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-socket" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.914978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.914545 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-data" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.914978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.914560 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804d523e-4a52-4960-9564-912a704258d2-istiod-ca-cert" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:38.914978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.914580 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-credential-socket" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.914978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.914599 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-certs" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915006 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twsfd\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-kube-api-access-twsfd\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915024 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915040 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915052 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915065 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915078 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915092 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915103 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106de83-4886-4b14-815a-6e8fb28d9621-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915115 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915128 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915139 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915151 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915164 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915176 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915190 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915202 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202a9b97-fc86-4e8d-b3e8-380942de9f32-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915214 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915226 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915237 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106de83-4886-4b14-815a-6e8fb28d9621-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915248 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915260 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpf6r\" (UniqueName: \"kubernetes.io/projected/202a9b97-fc86-4e8d-b3e8-380942de9f32-kube-api-access-wpf6r\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915271 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915284 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915295 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106de83-4886-4b14-815a-6e8fb28d9621-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915306 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915317 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915329 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d523e-4a52-4960-9564-912a704258d2-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915341 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202a9b97-fc86-4e8d-b3e8-380942de9f32-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915352 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbz8\" (UniqueName: \"kubernetes.io/projected/1106de83-4886-4b14-815a-6e8fb28d9621-kube-api-access-vgbz8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.915355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.915366 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:38.917747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.917711 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/804d523e-4a52-4960-9564-912a704258d2-istio-podinfo" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:38.918034 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.918008 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-kube-api-access-cvrss" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "kube-api-access-cvrss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:38.918488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.918458 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-envoy" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:38.918752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:38.918716 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-istio-token" pod "804d523e-4a52-4960-9564-912a704258d2" (UID: "804d523e-4a52-4960-9564-912a704258d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:39.017004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.016966 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvrss\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-kube-api-access-cvrss\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:39.017004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.016998 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d523e-4a52-4960-9564-912a704258d2-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:39.017206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.017011 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d523e-4a52-4960-9564-912a704258d2-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:39.017206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.017023 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d523e-4a52-4960-9564-912a704258d2-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:39.204511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.204471 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948f189d-119b-4657-ab78-81373fffd7e3" path="/var/lib/kubelet/pods/948f189d-119b-4657-ab78-81373fffd7e3/volumes" Oct 30 01:03:39.255712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.255676 4905 patch_prober.go:29] interesting pod/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:03:39.255854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.255738 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:03:39.414623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.414544 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" Oct 30 01:03:39.414623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.414563 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns" event={"ID":"202a9b97-fc86-4e8d-b3e8-380942de9f32","Type":"ContainerDied","Data":"4fbc5e7c6f917ada31f62422f2d5c87b492d2a6bfa5ea01bbc7b7cb18dca19ce"} Oct 30 01:03:39.414819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.414636 4905 scope.go:122] "RemoveContainer" containerID="27782b6555faa15ee8fc3fc11661f4a46f5e309bf03c420ccfcb7c07b93ee8fc" Oct 30 01:03:39.416746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.416721 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" Oct 30 01:03:39.417772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.416748 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" event={"ID":"1106de83-4886-4b14-815a-6e8fb28d9621","Type":"ContainerDied","Data":"96d0f770c416e8bcd24b572dc542df2acce4df738c9d23e9cadf6292304d6804"} Oct 30 01:03:39.419020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.419000 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" Oct 30 01:03:39.419141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.419020 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx" event={"ID":"804d523e-4a52-4960-9564-912a704258d2","Type":"ContainerDied","Data":"2f424e2fc92d1fdeb1e6125b8058c93ecf0cec4c607888bf773220066ce6ed4e"} Oct 30 01:03:39.420663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.420643 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" Oct 30 01:03:39.420772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.420660 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x" event={"ID":"d021244d-d6b8-4390-b1a1-5aafeaf6d0b1","Type":"ContainerDied","Data":"ab622b5808beb878ab4f90af4038f0bd5ea061e97896c2086e15af8cc2a89998"} Oct 30 01:03:39.424326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.424306 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" event={"ID":"d3d11eb5-43ca-41a9-b945-3219c8e0bcc7","Type":"ContainerDied","Data":"300bcbf233dc808445ba96a2fe247fa213db12fd8179e1df4a1e6fbe53b14786"} Oct 30 01:03:39.424432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.424335 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz" Oct 30 01:03:39.438550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.438507 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" event={"ID":"8445bf3a-7f07-4bc5-a092-3eef8b985999","Type":"ContainerDied","Data":"1044f63f3d42697d33fd7f3a917f778197b4fe601fea471d0904c73681346b7f"} Oct 30 01:03:39.438755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.438739 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r" Oct 30 01:03:39.441332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.441314 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" event={"ID":"22737c92-facc-49a0-b4a2-4980aacd0c36","Type":"ContainerDied","Data":"4f6e67db69a6e0a00eda25d50301313fb0d6af544cca69d7497507f54d9c851f"} Oct 30 01:03:39.441545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.441480 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd" Oct 30 01:03:39.451631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.449729 4905 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:03:39.451631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.449774 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:03:39.456760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.456721 4905 scope.go:122] "RemoveContainer" containerID="d2fcd78cc64d7106d9603657fb66163aabd6c179a97144bf049db278b1b5dfdc" Oct 30 01:03:39.504862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.504843 4905 scope.go:122] "RemoveContainer" containerID="e4e39fef92c046edfd69bf920a93b24e55e092a3eebebaa7685a1ea90b304df1" Oct 30 01:03:39.518755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.518647 4905 scope.go:122] "RemoveContainer" containerID="59541bda9b9e3d0bd462ac76ed3b22a3f6a50fb4fd1bc537fa29ffdb470eb0dd" Oct 30 01:03:39.519162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.519145 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x"] Oct 30 01:03:39.535179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.535155 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-lfn5x"] Oct 30 01:03:39.561428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.561407 4905 scope.go:122] "RemoveContainer" containerID="fa08e786d89b46ab1ef9ec92b87cf7a1aca1fff96678132bbd1ba4dece0a4210" Oct 30 01:03:39.565504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.565482 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz"] Oct 30 01:03:39.572325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.572301 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-5qspz"] Oct 30 01:03:39.575026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.575005 4905 scope.go:122] "RemoveContainer" containerID="2446d6b5e1ce21c5a20a13f4eecdf575b8f556d28cc7f5fc8046dc278f446641" Oct 30 01:03:39.586082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.586053 4905 scope.go:122] "RemoveContainer" containerID="1664e06f01c5c5f3275a7d7a7f67ae122889c5dca64e8f4442ad29c836e5b7c8" Oct 30 01:03:39.612371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.612312 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns"] Oct 30 01:03:39.621270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.621225 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-hvtns"] Oct 30 01:03:39.703674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.703585 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx"] Oct 30 01:03:39.711070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.711034 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fc4hx"] Oct 30 01:03:39.720738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.719824 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd"] Oct 30 01:03:39.735648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.735605 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-cbznd"] Oct 30 01:03:39.774615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.771956 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz"] Oct 30 01:03:39.778157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.778130 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-fnxsz"] Oct 30 01:03:39.801587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.801561 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r"] Oct 30 01:03:39.807575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:39.807550 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-vmk6r"] Oct 30 01:03:40.241541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:40.235695 4905 patch_prober.go:29] interesting pod/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:03:40.241541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:40.235751 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:03:40.792327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:40.792288 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" podUID="84b515ec-60fb-4b04-b97c-57141b253fb3" containerName="payload-pre-processing" containerID="cri-o://dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58" gracePeriod=25 Oct 30 01:03:41.214487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.214455 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" path="/var/lib/kubelet/pods/1106de83-4886-4b14-815a-6e8fb28d9621/volumes" Oct 30 01:03:41.214811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.214580 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:03:41.215635 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.215600 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" path="/var/lib/kubelet/pods/202a9b97-fc86-4e8d-b3e8-380942de9f32/volumes" Oct 30 01:03:41.216238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.216209 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" path="/var/lib/kubelet/pods/22737c92-facc-49a0-b4a2-4980aacd0c36/volumes" Oct 30 01:03:41.217309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.217287 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804d523e-4a52-4960-9564-912a704258d2" path="/var/lib/kubelet/pods/804d523e-4a52-4960-9564-912a704258d2/volumes" Oct 30 01:03:41.217996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.217976 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" path="/var/lib/kubelet/pods/8445bf3a-7f07-4bc5-a092-3eef8b985999/volumes" Oct 30 01:03:41.218622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.218603 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" path="/var/lib/kubelet/pods/d021244d-d6b8-4390-b1a1-5aafeaf6d0b1/volumes" Oct 30 01:03:41.219631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.219613 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" path="/var/lib/kubelet/pods/d3d11eb5-43ca-41a9-b945-3219c8e0bcc7/volumes" Oct 30 01:03:41.255856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.255625 4905 patch_prober.go:29] interesting pod/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:03:41.255856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.255691 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:03:41.275926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.273350 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b515ec-60fb-4b04-b97c-57141b253fb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b515ec-60fb-4b04-b97c-57141b253fb3-plugins-config-volume\") pod \"84b515ec-60fb-4b04-b97c-57141b253fb3\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " Oct 30 01:03:41.275926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.273485 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b515ec-60fb-4b04-b97c-57141b253fb3-kube-api-access-bqz5d\" (UniqueName: \"kubernetes.io/projected/84b515ec-60fb-4b04-b97c-57141b253fb3-kube-api-access-bqz5d\") pod \"84b515ec-60fb-4b04-b97c-57141b253fb3\" (UID: \"84b515ec-60fb-4b04-b97c-57141b253fb3\") " Oct 30 01:03:41.275926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.273870 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b515ec-60fb-4b04-b97c-57141b253fb3-plugins-config-volume" pod "84b515ec-60fb-4b04-b97c-57141b253fb3" (UID: "84b515ec-60fb-4b04-b97c-57141b253fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:41.275926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.274437 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b515ec-60fb-4b04-b97c-57141b253fb3-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:41.281710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.280769 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b515ec-60fb-4b04-b97c-57141b253fb3-kube-api-access-bqz5d" pod "84b515ec-60fb-4b04-b97c-57141b253fb3" (UID: "84b515ec-60fb-4b04-b97c-57141b253fb3"). InnerVolumeSpecName "kube-api-access-bqz5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:41.375735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.375705 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqz5d\" (UniqueName: \"kubernetes.io/projected/84b515ec-60fb-4b04-b97c-57141b253fb3-kube-api-access-bqz5d\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:41.478782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.478697 4905 generic.go:356] "Generic (PLEG): container finished" podID="84b515ec-60fb-4b04-b97c-57141b253fb3" containerID="dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58" exitCode=0 Oct 30 01:03:41.478782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.478746 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" Oct 30 01:03:41.479040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.478777 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" event={"ID":"84b515ec-60fb-4b04-b97c-57141b253fb3","Type":"ContainerDied","Data":"dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58"} Oct 30 01:03:41.479040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.478828 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m" event={"ID":"84b515ec-60fb-4b04-b97c-57141b253fb3","Type":"ContainerDied","Data":"6d9d23af3fdf4a3ec59c5f6d2db9ebc48be850a18875da57e36b5e7c35c8ce97"} Oct 30 01:03:41.479040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.478851 4905 scope.go:122] "RemoveContainer" containerID="dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58" Oct 30 01:03:41.496976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.496956 4905 scope.go:122] "RemoveContainer" containerID="dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58" Oct 30 01:03:41.497486 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:41.497459 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58\": container with ID starting with dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58 not found: ID does not exist" containerID="dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58" Oct 30 01:03:41.497670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.497495 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58"} err="failed to get container status \"dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58\": rpc error: code = NotFound desc = could not find container \"dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58\": container with ID starting with dcba73307b2fef22989b3e3cda45ba6dbc48ba020ada056bf672c39f9587bf58 not found: ID does not exist" Oct 30 01:03:41.648453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.592882 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m"] Oct 30 01:03:41.648453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.645170 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b814fb-856b8d8dcf-bm94m"] Oct 30 01:03:41.823629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:41.823604 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:03:42.209088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.209011 4905 scope.go:122] "RemoveContainer" containerID="fb10886d88e2025843d16183dfced3812bd7578d698231fe0b513ba7cf05310c" Oct 30 01:03:42.221465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.221448 4905 scope.go:122] "RemoveContainer" containerID="575b734ce5a6c01c9bbdabf5165a8cc5510502d42d10d5672337c7273740b7df" Oct 30 01:03:42.233930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.233905 4905 patch_prober.go:29] interesting pod/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:03:42.234144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.234117 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:03:42.259655 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.259633 4905 scope.go:122] "RemoveContainer" containerID="545a37e73956389778854390d30df7881833ee68384ab15cb1823d8888e166e5" Oct 30 01:03:42.270994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.270973 4905 scope.go:122] "RemoveContainer" containerID="89354f0a7f28db92b341ef60b8a019e3e4ff0c002d3ffa16e55cad967b2abfc0" Oct 30 01:03:42.281603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.281583 4905 scope.go:122] "RemoveContainer" containerID="ce80954b4b49ff11a5f0b59f56a3bbc2a98692b012796dc81452a5c579cf7771" Oct 30 01:03:42.291958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.291943 4905 scope.go:122] "RemoveContainer" containerID="b44a9c50312d85cc8d19e13b0c0ea2c2b6f7d02e8f937b38c9df6dcd3e784f3a" Oct 30 01:03:42.303927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.303912 4905 scope.go:122] "RemoveContainer" containerID="c3ae2efd8e7c14d50ba9431bf897216fc57a6b2c0aa6e2cbad74a2075e7374bb" Oct 30 01:03:42.314561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.314498 4905 scope.go:122] "RemoveContainer" containerID="a3530b4f43b671ec68880117cacc5802bbef964591ebc3620945aa5c14d9ed87" Oct 30 01:03:42.324392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.324373 4905 scope.go:122] "RemoveContainer" containerID="ca338d41ec01333597ea7740f0e062040df218cd478458dce45690f41ab0cdd9" Oct 30 01:03:42.367880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.367860 4905 scope.go:122] "RemoveContainer" containerID="aa4029121dedbca3c03222c09c4e406dad67c84f6c74a01d83abab73323e67b2" Oct 30 01:03:42.378491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.378472 4905 scope.go:122] "RemoveContainer" containerID="f0e71122fff2db57e8969b481f2e3548ca0f310d28f60d082e2a55d1c4db7064" Oct 30 01:03:42.390761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.388663 4905 scope.go:122] "RemoveContainer" containerID="116327becb15668b34ec3bf8c501e3f469f460058efdb2c8d639202aec573f58" Oct 30 01:03:42.441897 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:42.441863 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:03:42.839037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:42.839001 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:03:43.280557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:43.269268 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b515ec-60fb-4b04-b97c-57141b253fb3" path="/var/lib/kubelet/pods/84b515ec-60fb-4b04-b97c-57141b253fb3/volumes" Oct 30 01:03:43.447544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:43.424585 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:03:43.448851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:43.448495 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:03:43.489816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:43.489782 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:03:43.808141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:43.808104 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4"] Oct 30 01:03:43.813556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:43.813488 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fcc0d3e0-5scd4"] Oct 30 01:03:45.238466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:45.238424 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38797784-bbdc-42d1-9bbb-02baac6b96ac" path="/var/lib/kubelet/pods/38797784-bbdc-42d1-9bbb-02baac6b96ac/volumes" Oct 30 01:03:47.026562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.026503 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8"] Oct 30 01:03:47.027057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.026741 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" containerID="cri-o://3ec30f8bc037bcc95e6b06cdf6d5bd0fd182c07ac744b4402f9bd54a1c069ebf" gracePeriod=30 Oct 30 01:03:47.040066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.040036 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn"] Oct 30 01:03:47.040297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.040273 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" containerID="cri-o://45a3e79d4a6cf0cb9420cdda1ae452e7894d087f35c7203f18c216f044161f6e" gracePeriod=30 Oct 30 01:03:47.045744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.045719 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr"] Oct 30 01:03:47.045906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.045887 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" containerID="cri-o://1aba5411c9c58a0f8d8a8e25d2ed661938822ff93a07d3945da37ffc73878990" gracePeriod=30 Oct 30 01:03:47.055004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.054968 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s"] Oct 30 01:03:47.055258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.055196 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" containerID="cri-o://9e6cb3b614c2fc7efdcd39fed7900a943ca82188078f66cc84d613603ed2c3e5" gracePeriod=30 Oct 30 01:03:47.372994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.372906 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:03:47.372994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:47.372970 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:03:50.482379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:50.482323 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh"] Oct 30 01:03:50.482908 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:50.482634 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" containerID="cri-o://ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3" gracePeriod=30 Oct 30 01:03:51.926270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.926232 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth"] Oct 30 01:03:51.940273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.940245 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" Oct 30 01:03:51.940548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.940512 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" Oct 30 01:03:51.940662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.940651 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" Oct 30 01:03:51.940742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.940730 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" Oct 30 01:03:51.940819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.940809 4905 cpu_manager.go:414] "containerMap: removing container" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" Oct 30 01:03:51.940919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.940909 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" Oct 30 01:03:51.941001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.940990 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" Oct 30 01:03:51.941084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.941074 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" Oct 30 01:03:51.941959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.941942 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" Oct 30 01:03:51.942077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942065 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" Oct 30 01:03:51.942179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942169 4905 cpu_manager.go:414] "containerMap: removing container" podUID="84b515ec-60fb-4b04-b97c-57141b253fb3" containerName="payload-pre-processing" Oct 30 01:03:51.942257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942246 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b515ec-60fb-4b04-b97c-57141b253fb3" containerName="payload-pre-processing" Oct 30 01:03:51.942355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942344 4905 cpu_manager.go:414] "containerMap: removing container" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" Oct 30 01:03:51.942439 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942429 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" Oct 30 01:03:51.942544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942534 4905 cpu_manager.go:414] "containerMap: removing container" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" Oct 30 01:03:51.942621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942610 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" Oct 30 01:03:51.942713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.942703 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fdc88044-2936-45ab-a3a1-9bb59e3267a8" containerName="revoke-keys" Oct 30 01:03:51.961722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.961689 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc88044-2936-45ab-a3a1-9bb59e3267a8" containerName="revoke-keys" Oct 30 01:03:51.962085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.962066 4905 cpu_manager.go:414] "containerMap: removing container" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" Oct 30 01:03:51.962215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.962203 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" Oct 30 01:03:51.962632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.962614 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="948f189d-119b-4657-ab78-81373fffd7e3" containerName="istio-proxy" Oct 30 01:03:51.962750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.962740 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" Oct 30 01:03:51.962849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.962839 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" Oct 30 01:03:51.962932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.962923 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8445bf3a-7f07-4bc5-a092-3eef8b985999" containerName="istio-proxy" Oct 30 01:03:51.963004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.962996 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1106de83-4886-4b14-815a-6e8fb28d9621" containerName="istio-proxy" Oct 30 01:03:51.963078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.963068 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="804d523e-4a52-4960-9564-912a704258d2" containerName="istio-proxy" Oct 30 01:03:51.963161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.963152 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" Oct 30 01:03:51.963242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.963233 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" Oct 30 01:03:51.963323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.963314 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b515ec-60fb-4b04-b97c-57141b253fb3" containerName="payload-pre-processing" Oct 30 01:03:51.963416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.963406 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="202a9b97-fc86-4e8d-b3e8-380942de9f32" containerName="istio-proxy" Oct 30 01:03:51.963493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.963484 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="22737c92-facc-49a0-b4a2-4980aacd0c36" containerName="istio-proxy" Oct 30 01:03:51.963614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:51.963602 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc88044-2936-45ab-a3a1-9bb59e3267a8" containerName="revoke-keys" Oct 30 01:03:52.015732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.014757 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth"] Oct 30 01:03:52.015732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.014886 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.170570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.152427 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c49c0c-openshift-default-dockercfg-7lzsg\"" Oct 30 01:03:52.170570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.155113 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-credential-socket\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.170570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.155177 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-token\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.170570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.155224 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-socket\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.170570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.155367 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-certs\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.170570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.155417 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istiod-ca-cert\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.170570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.155497 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-envoy\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.262245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.252620 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-data\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.262245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.252705 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-podinfo\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.262245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.252788 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbp4\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-kube-api-access-5jbp4\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.366924 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-socket\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367115 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-certs\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367171 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istiod-ca-cert\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367248 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-envoy\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367346 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-data\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367401 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-podinfo\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367465 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbp4\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-kube-api-access-5jbp4\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367551 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-credential-socket\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.369118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.367607 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-token\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.375225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.375196 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-data\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.375694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.375672 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-certs\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.376533 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.376490 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istiod-ca-cert\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.378154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.378128 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-credential-socket\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.427598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.400627 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbp4\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-kube-api-access-5jbp4\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.427598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.405960 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-token\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.555145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.510805 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-envoy\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.555145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.516334 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-socket\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.566592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.563325 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-podinfo\") pod \"e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.591577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.584051 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:52.876229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.875973 4905 generic.go:356] "Generic (PLEG): container finished" podID="08147324-90f2-4d15-902c-c2e0d45eed52" containerID="1aba5411c9c58a0f8d8a8e25d2ed661938822ff93a07d3945da37ffc73878990" exitCode=0 Oct 30 01:03:52.876229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.876157 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" event={"ID":"08147324-90f2-4d15-902c-c2e0d45eed52","Type":"ContainerDied","Data":"1aba5411c9c58a0f8d8a8e25d2ed661938822ff93a07d3945da37ffc73878990"} Oct 30 01:03:52.881743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.881717 4905 generic.go:356] "Generic (PLEG): container finished" podID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerID="9e6cb3b614c2fc7efdcd39fed7900a943ca82188078f66cc84d613603ed2c3e5" exitCode=0 Oct 30 01:03:52.881971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.881837 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" event={"ID":"c33408e5-7fea-4575-aabb-09bee5bbb41e","Type":"ContainerDied","Data":"9e6cb3b614c2fc7efdcd39fed7900a943ca82188078f66cc84d613603ed2c3e5"} Oct 30 01:03:52.882110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.882099 4905 scope.go:122] "RemoveContainer" containerID="61888bfb77b2d6031d167ef0ff5916473beb626ce0f1a8dc5221e44efaf3b1e9" Oct 30 01:03:52.900274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.899750 4905 generic.go:356] "Generic (PLEG): container finished" podID="520fcf61-f470-433f-8767-58d971b4e9a9" containerID="3ec30f8bc037bcc95e6b06cdf6d5bd0fd182c07ac744b4402f9bd54a1c069ebf" exitCode=0 Oct 30 01:03:52.900274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.899857 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" event={"ID":"520fcf61-f470-433f-8767-58d971b4e9a9","Type":"ContainerDied","Data":"3ec30f8bc037bcc95e6b06cdf6d5bd0fd182c07ac744b4402f9bd54a1c069ebf"} Oct 30 01:03:52.902482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.902288 4905 generic.go:356] "Generic (PLEG): container finished" podID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerID="45a3e79d4a6cf0cb9420cdda1ae452e7894d087f35c7203f18c216f044161f6e" exitCode=0 Oct 30 01:03:52.902482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.902427 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" event={"ID":"4d4b60c4-d140-43ee-9c61-802cba188d6b","Type":"ContainerDied","Data":"45a3e79d4a6cf0cb9420cdda1ae452e7894d087f35c7203f18c216f044161f6e"} Oct 30 01:03:52.991832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:52.991801 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.089374 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-podinfo\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.089441 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-credential-socket\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.089568 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4b60c4-d140-43ee-9c61-802cba188d6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4b60c4-d140-43ee-9c61-802cba188d6b-istiod-ca-cert\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.089774 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-kube-api-access-4jkr5\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-kube-api-access-4jkr5\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.089823 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-socket\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.089879 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-envoy\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.090053 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-certs\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.090185 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-token\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-token\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.090260 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-data\") pod \"4d4b60c4-d140-43ee-9c61-802cba188d6b\" (UID: \"4d4b60c4-d140-43ee-9c61-802cba188d6b\") " Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.091744 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-data" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.100201 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4b60c4-d140-43ee-9c61-802cba188d6b-istiod-ca-cert" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.102598 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-socket" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.110606 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-credential-socket" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.165855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.110733 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-certs" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.208757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.197651 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.208757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.197684 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.208757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.197697 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.208757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.197711 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4b60c4-d140-43ee-9c61-802cba188d6b-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.208757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.197725 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.222877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.209731 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-kube-api-access-4jkr5" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "kube-api-access-4jkr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:53.222877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.214123 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-envoy" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.222877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.218645 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-podinfo" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:53.222877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.220705 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-token" pod "4d4b60c4-d140-43ee-9c61-802cba188d6b" (UID: "4d4b60c4-d140-43ee-9c61-802cba188d6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:53.331265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.331235 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.333248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.333186 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jkr5\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-kube-api-access-4jkr5\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.333398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.333385 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.333488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.333478 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4b60c4-d140-43ee-9c61-802cba188d6b-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.615227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.604019 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 01:03:53.721131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.721098 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-credential-socket\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.721417 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.721380 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-data\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.721641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.721598 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-podinfo\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.722316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.722273 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-kube-api-access-jwgdk\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-kube-api-access-jwgdk\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.722763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.722685 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-certs\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.725067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.725044 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33408e5-7fea-4575-aabb-09bee5bbb41e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33408e5-7fea-4575-aabb-09bee5bbb41e-istiod-ca-cert\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.725338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.725306 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-token\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-token\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.726001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.725984 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-envoy\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.726272 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.726257 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-socket\") pod \"c33408e5-7fea-4575-aabb-09bee5bbb41e\" (UID: \"c33408e5-7fea-4575-aabb-09bee5bbb41e\") " Oct 30 01:03:53.736057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.723632 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-certs" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.736057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.723792 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-credential-socket" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.736057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.723948 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-data" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.736057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.735738 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33408e5-7fea-4575-aabb-09bee5bbb41e-istiod-ca-cert" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:53.782557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.769903 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth"] Oct 30 01:03:53.784828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.784794 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-socket" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.852624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.833607 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-token" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:53.852940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.851631 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-envoy" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:53.853046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.851340 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq"] Oct 30 01:03:53.853173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.853160 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7452d0-mc8zq"] Oct 30 01:03:53.853270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.847485 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.853384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.853363 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.853482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.853465 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.853600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.853587 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.871257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.871236 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33408e5-7fea-4575-aabb-09bee5bbb41e-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.871358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.866798 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:53.871554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.871502 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:53.871728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.871705 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:03:53.975581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.975491 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-kube-api-access-jwgdk" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "kube-api-access-jwgdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:53.975708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.975602 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-podinfo" pod "c33408e5-7fea-4575-aabb-09bee5bbb41e" (UID: "c33408e5-7fea-4575-aabb-09bee5bbb41e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:53.979150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.979063 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.979150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.979092 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgdk\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-kube-api-access-jwgdk\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.979150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.979107 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.979150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.979122 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33408e5-7fea-4575-aabb-09bee5bbb41e-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.986413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.984874 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" event={"ID":"08147324-90f2-4d15-902c-c2e0d45eed52","Type":"ContainerDied","Data":"f22b7317c18571927f86c443e4722ffc1aabb474539aa66d16e7bb4f43e0d74d"} Oct 30 01:03:53.986413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.984938 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f22b7317c18571927f86c443e4722ffc1aabb474539aa66d16e7bb4f43e0d74d" Oct 30 01:03:53.986413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.985233 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 01:03:53.992660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.988825 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 01:03:53.992660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.991179 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" event={"ID":"c33408e5-7fea-4575-aabb-09bee5bbb41e","Type":"ContainerDied","Data":"b327aed0b4b1230669513c92eed8979e5f61657dca7eb8f9c211393f158aabe0"} Oct 30 01:03:53.992660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.991234 4905 scope.go:122] "RemoveContainer" containerID="9e6cb3b614c2fc7efdcd39fed7900a943ca82188078f66cc84d613603ed2c3e5" Oct 30 01:03:53.992660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:53.991331 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s" Oct 30 01:03:54.012608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.012509 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" event={"ID":"fffa3510-3a48-4aff-b299-584ce5e7a0ea","Type":"ContainerStarted","Data":"58320035bb7245ce123401831198fbe578f71b951fa3a5142bd59dc5aaf7eb13"} Oct 30 01:03:54.015024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.014749 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" event={"ID":"520fcf61-f470-433f-8767-58d971b4e9a9","Type":"ContainerDied","Data":"cf06b3b37546ebf9d2cdd15e5b04792f9eaf73c511ff604e7ed1d09178d2c504"} Oct 30 01:03:54.015024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.014841 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" Oct 30 01:03:54.018544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.017452 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" event={"ID":"4d4b60c4-d140-43ee-9c61-802cba188d6b","Type":"ContainerDied","Data":"ab5bfa7f23231330fcad2ec2726098fc3d7559d9fa9d90e068cd33392f4a3c1a"} Oct 30 01:03:54.018544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.017592 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn" Oct 30 01:03:54.082287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082022 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/520fcf61-f470-433f-8767-58d971b4e9a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520fcf61-f470-433f-8767-58d971b4e9a9-istio-podinfo\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.082287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082088 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08147324-90f2-4d15-902c-c2e0d45eed52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08147324-90f2-4d15-902c-c2e0d45eed52-istiod-ca-cert\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.082287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082141 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-kube-api-access-kt26s\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-kube-api-access-kt26s\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.082287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082184 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-socket\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.082755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082313 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-socket\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.082755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082406 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-credential-socket\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.082755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082507 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-certs\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.082755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082566 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-envoy\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.082755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082607 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-istio-token\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-istio-token\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.082755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082668 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-envoy\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.083056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082773 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-credential-socket\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.083056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082849 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-kube-api-access-gctzt\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-kube-api-access-gctzt\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.083056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082899 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520fcf61-f470-433f-8767-58d971b4e9a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520fcf61-f470-433f-8767-58d971b4e9a9-istiod-ca-cert\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.083056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082957 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-certs\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.083056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.082984 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-data\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.083056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.083039 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-istio-token\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-istio-token\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.084220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.083069 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-data\") pod \"520fcf61-f470-433f-8767-58d971b4e9a9\" (UID: \"520fcf61-f470-433f-8767-58d971b4e9a9\") " Oct 30 01:03:54.084220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.083178 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08147324-90f2-4d15-902c-c2e0d45eed52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08147324-90f2-4d15-902c-c2e0d45eed52-istio-podinfo\") pod \"08147324-90f2-4d15-902c-c2e0d45eed52\" (UID: \"08147324-90f2-4d15-902c-c2e0d45eed52\") " Oct 30 01:03:54.088505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.088182 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08147324-90f2-4d15-902c-c2e0d45eed52-istio-podinfo" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:54.090619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.090405 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/520fcf61-f470-433f-8767-58d971b4e9a9-istio-podinfo" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:54.090988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.090792 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08147324-90f2-4d15-902c-c2e0d45eed52-istiod-ca-cert" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:54.093359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.093160 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-kube-api-access-kt26s" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "kube-api-access-kt26s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:54.093699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.093490 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-socket" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.093998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.093835 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-socket" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.094257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.094109 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-credential-socket" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.094548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.094376 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-certs" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.096968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.096705 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-envoy" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.099314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.099070 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-credential-socket" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.100728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.100478 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520fcf61-f470-433f-8767-58d971b4e9a9-istiod-ca-cert" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:54.101025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.100870 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-certs" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.102592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.101197 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-data" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.106887 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.103008 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-envoy" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.106887 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.103009 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-kube-api-access-gctzt" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "kube-api-access-gctzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:54.106887 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.103059 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-istio-token" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:54.106887 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.106584 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-data" pod "520fcf61-f470-433f-8767-58d971b4e9a9" (UID: "520fcf61-f470-433f-8767-58d971b4e9a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:54.109404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.109367 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-istio-token" pod "08147324-90f2-4d15-902c-c2e0d45eed52" (UID: "08147324-90f2-4d15-902c-c2e0d45eed52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:54.182812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.182772 4905 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:03:54.183122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.183088 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:03:54.186632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.186475 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctzt\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-kube-api-access-gctzt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.186786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.186768 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520fcf61-f470-433f-8767-58d971b4e9a9-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.186883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.186872 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.186990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.186978 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187069 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187163 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187243 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08147324-90f2-4d15-902c-c2e0d45eed52-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187335 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520fcf61-f470-433f-8767-58d971b4e9a9-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187429 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08147324-90f2-4d15-902c-c2e0d45eed52-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187533 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt26s\" (UniqueName: \"kubernetes.io/projected/08147324-90f2-4d15-902c-c2e0d45eed52-kube-api-access-kt26s\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187628 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187707 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187806 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187884 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.187985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.187975 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08147324-90f2-4d15-902c-c2e0d45eed52-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.188068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.188058 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520fcf61-f470-433f-8767-58d971b4e9a9-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.188145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.188135 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.188220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.188210 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520fcf61-f470-433f-8767-58d971b4e9a9-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:54.258779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.247874 4905 scope.go:122] "RemoveContainer" containerID="3ec30f8bc037bcc95e6b06cdf6d5bd0fd182c07ac744b4402f9bd54a1c069ebf" Oct 30 01:03:54.300171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.300140 4905 patch_prober.go:29] interesting pod/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:03:54.300361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.300339 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:03:54.377608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.377053 4905 scope.go:122] "RemoveContainer" containerID="45a3e79d4a6cf0cb9420cdda1ae452e7894d087f35c7203f18c216f044161f6e" Oct 30 01:03:54.390593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.389155 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s"] Oct 30 01:03:54.398028 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.398001 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4274s"] Oct 30 01:03:54.453873 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.453836 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn"] Oct 30 01:03:54.458171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.458139 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-95bbn"] Oct 30 01:03:54.486557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.486291 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8"] Oct 30 01:03:54.491435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:54.491412 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-zr2d8"] Oct 30 01:03:55.062059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.062025 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" event={"ID":"fffa3510-3a48-4aff-b299-584ce5e7a0ea","Type":"ContainerStarted","Data":"db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42"} Oct 30 01:03:55.065118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.065094 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr" Oct 30 01:03:55.109487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.109424 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podStartSLOduration=4.109406216 podStartE2EDuration="4.109406216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:55.108421634 +0000 UTC m=+1520.051594825" watchObservedRunningTime="2026-10-30 01:03:55.109406216 +0000 UTC m=+1520.052579408" Oct 30 01:03:55.161334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.161299 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr"] Oct 30 01:03:55.165868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.165840 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-4cpvr"] Oct 30 01:03:55.166746 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:55.166716 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08147324_90f2_4d15_902c_c2e0d45eed52.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08147324_90f2_4d15_902c_c2e0d45eed52.slice/crio-f22b7317c18571927f86c443e4722ffc1aabb474539aa66d16e7bb4f43e0d74d\": RecentStats: unable to find data in memory cache]" Oct 30 01:03:55.194391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.194369 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" path="/var/lib/kubelet/pods/08147324-90f2-4d15-902c-c2e0d45eed52/volumes" Oct 30 01:03:55.194870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.194856 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" path="/var/lib/kubelet/pods/4d4b60c4-d140-43ee-9c61-802cba188d6b/volumes" Oct 30 01:03:55.195303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.195290 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" path="/var/lib/kubelet/pods/520fcf61-f470-433f-8767-58d971b4e9a9/volumes" Oct 30 01:03:55.196099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.196078 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" path="/var/lib/kubelet/pods/c33408e5-7fea-4575-aabb-09bee5bbb41e/volumes" Oct 30 01:03:55.196833 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.196813 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e" path="/var/lib/kubelet/pods/ee4391c0-4118-4ab5-bddc-a4d0ccc4da4e/volumes" Oct 30 01:03:55.585128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.585095 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:55.592438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.592407 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:55.824094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.824071 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:03:55.880251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880213 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-data\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880308 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edc9348d-15c2-410a-bcd7-3781c7888192-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc9348d-15c2-410a-bcd7-3781c7888192-istio-podinfo\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880346 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-certs\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880410 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-credential-socket\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880581 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-socket\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880606 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-envoy\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880629 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-data" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880744 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-istio-token\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-istio-token\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880804 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-certs" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880818 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-kube-api-access-7s59g\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-kube-api-access-7s59g\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880890 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc9348d-15c2-410a-bcd7-3781c7888192-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc9348d-15c2-410a-bcd7-3781c7888192-istiod-ca-cert\") pod \"edc9348d-15c2-410a-bcd7-3781c7888192\" (UID: \"edc9348d-15c2-410a-bcd7-3781c7888192\") " Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880892 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-socket" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.880913 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-credential-socket" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.881277 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc9348d-15c2-410a-bcd7-3781c7888192-istiod-ca-cert" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.881791 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.881814 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.881829 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.881843 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc9348d-15c2-410a-bcd7-3781c7888192-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.882823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.881858 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.884981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.884937 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-kube-api-access-7s59g" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "kube-api-access-7s59g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:55.885108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.885067 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-envoy" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:55.885108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.885094 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edc9348d-15c2-410a-bcd7-3781c7888192-istio-podinfo" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:55.886900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.886880 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-istio-token" pod "edc9348d-15c2-410a-bcd7-3781c7888192" (UID: "edc9348d-15c2-410a-bcd7-3781c7888192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:55.983448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.983416 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.983448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.983446 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s59g\" (UniqueName: \"kubernetes.io/projected/edc9348d-15c2-410a-bcd7-3781c7888192-kube-api-access-7s59g\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.983448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.983455 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc9348d-15c2-410a-bcd7-3781c7888192-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:55.983667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:55.983465 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc9348d-15c2-410a-bcd7-3781c7888192-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:03:56.075437 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.075353 4905 generic.go:356] "Generic (PLEG): container finished" podID="edc9348d-15c2-410a-bcd7-3781c7888192" containerID="ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3" exitCode=0 Oct 30 01:03:56.075922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.075427 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" event={"ID":"edc9348d-15c2-410a-bcd7-3781c7888192","Type":"ContainerDied","Data":"ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3"} Oct 30 01:03:56.075922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.075466 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" Oct 30 01:03:56.075922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.075489 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh" event={"ID":"edc9348d-15c2-410a-bcd7-3781c7888192","Type":"ContainerDied","Data":"2192928904ea9d5c2a3f6564b22e36b775903dc790d05605f565a3f1a5341515"} Oct 30 01:03:56.075922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.075501 4905 scope.go:122] "RemoveContainer" containerID="ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3" Oct 30 01:03:56.076137 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.076042 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:56.076876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.076861 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:03:56.088103 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.088089 4905 scope.go:122] "RemoveContainer" containerID="5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0" Oct 30 01:03:56.098029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.098013 4905 scope.go:122] "RemoveContainer" containerID="ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3" Oct 30 01:03:56.098275 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:56.098258 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3\": container with ID starting with ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3 not found: ID does not exist" containerID="ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3" Oct 30 01:03:56.098349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.098282 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3"} err="failed to get container status \"ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3\": rpc error: code = NotFound desc = could not find container \"ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3\": container with ID starting with ca5327c58357e6c94da1c97f6f99a4dceadf06203b1466506d547f3cc9a899f3 not found: ID does not exist" Oct 30 01:03:56.098349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.098298 4905 scope.go:122] "RemoveContainer" containerID="5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0" Oct 30 01:03:56.098552 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:03:56.098499 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0\": container with ID starting with 5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0 not found: ID does not exist" containerID="5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0" Oct 30 01:03:56.098552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.098536 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0"} err="failed to get container status \"5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0\": rpc error: code = NotFound desc = could not find container \"5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0\": container with ID starting with 5c1118ebaa1853acecd2790c985679a427382984fabf674f1a043a62512a02f0 not found: ID does not exist" Oct 30 01:03:56.191542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.191482 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh"] Oct 30 01:03:56.223768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:56.223738 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fcc0d3e0-openshift-default-57c6d99b89-l65vh"] Oct 30 01:03:57.193272 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:57.193239 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" path="/var/lib/kubelet/pods/edc9348d-15c2-410a-bcd7-3781c7888192/volumes" Oct 30 01:03:59.409750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.409713 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2"] Oct 30 01:03:59.410932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.410904 4905 cpu_manager.go:414] "containerMap: removing container" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" Oct 30 01:03:59.410932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.410928 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" Oct 30 01:03:59.411128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.410946 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" Oct 30 01:03:59.411128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.410954 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" Oct 30 01:03:59.411128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411080 4905 cpu_manager.go:414] "containerMap: removing container" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" Oct 30 01:03:59.411128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411088 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" Oct 30 01:03:59.411128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411118 4905 cpu_manager.go:414] "containerMap: removing container" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" Oct 30 01:03:59.411128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411125 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" Oct 30 01:03:59.411441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411162 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" Oct 30 01:03:59.411441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411169 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" Oct 30 01:03:59.411441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411193 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" Oct 30 01:03:59.411441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411200 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d11eb5-43ca-41a9-b945-3219c8e0bcc7" containerName="istio-proxy" Oct 30 01:03:59.411441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411213 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" Oct 30 01:03:59.411441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411220 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d021244d-d6b8-4390-b1a1-5aafeaf6d0b1" containerName="istio-proxy" Oct 30 01:03:59.411441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411441 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4b60c4-d140-43ee-9c61-802cba188d6b" containerName="istio-proxy" Oct 30 01:03:59.411835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411459 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" Oct 30 01:03:59.411835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411470 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" Oct 30 01:03:59.411835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411479 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" Oct 30 01:03:59.411835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411488 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="08147324-90f2-4d15-902c-c2e0d45eed52" containerName="istio-proxy" Oct 30 01:03:59.411835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.411497 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="520fcf61-f470-433f-8767-58d971b4e9a9" containerName="istio-proxy" Oct 30 01:03:59.417955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.417931 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:03:59.438346 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.438310 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2"] Oct 30 01:03:59.443433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.443409 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c49c0c\"" Oct 30 01:03:59.443656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.443409 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c49c0c-dockercfg-ml4bh\"" Oct 30 01:03:59.560543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.556891 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9d7\" (UniqueName: \"kubernetes.io/projected/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-kube-api-access-4q9d7\") pod \"payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:03:59.560543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.556961 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:03:59.687844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.686696 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9d7\" (UniqueName: \"kubernetes.io/projected/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-kube-api-access-4q9d7\") pod \"payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:03:59.687844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.686744 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:03:59.687844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.687483 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:03:59.804587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.796568 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp"] Oct 30 01:03:59.805397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.804908 4905 cpu_manager.go:414] "containerMap: removing container" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" Oct 30 01:03:59.805397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.804933 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc9348d-15c2-410a-bcd7-3781c7888192" containerName="istio-proxy" Oct 30 01:03:59.805397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.804960 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" Oct 30 01:03:59.805397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.804969 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" Oct 30 01:03:59.805397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.805195 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33408e5-7fea-4575-aabb-09bee5bbb41e" containerName="istio-proxy" Oct 30 01:03:59.826450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.826385 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp"] Oct 30 01:03:59.827955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.826545 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:03:59.840579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:03:59.840549 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9d7\" (UniqueName: \"kubernetes.io/projected/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-kube-api-access-4q9d7\") pod \"payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:04:00.025489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.025457 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2e8ebe-01db-458b-ad90-1622880f0629-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:00.025677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.025650 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8fr\" (UniqueName: \"kubernetes.io/projected/2a2e8ebe-01db-458b-ad90-1622880f0629-kube-api-access-gf8fr\") pod \"payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:00.038693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.038666 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:04:00.127338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.127310 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2e8ebe-01db-458b-ad90-1622880f0629-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:00.127508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.127386 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8fr\" (UniqueName: \"kubernetes.io/projected/2a2e8ebe-01db-458b-ad90-1622880f0629-kube-api-access-gf8fr\") pod \"payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:00.128062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.128034 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2e8ebe-01db-458b-ad90-1622880f0629-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:00.164002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.163157 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8fr\" (UniqueName: \"kubernetes.io/projected/2a2e8ebe-01db-458b-ad90-1622880f0629-kube-api-access-gf8fr\") pod \"payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:00.387660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.387625 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2"] Oct 30 01:04:00.454067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.454044 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:00.852598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:00.852470 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp"] Oct 30 01:04:01.292037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.291100 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" event={"ID":"bd292360-d2fd-4c2b-b6a1-208170d8c1f1","Type":"ContainerStarted","Data":"8d24bc5a3e419f1ff207240e6de1d0d864c1621b39fc544ae59f0842daae5c0c"} Oct 30 01:04:01.292037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.291158 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" event={"ID":"bd292360-d2fd-4c2b-b6a1-208170d8c1f1","Type":"ContainerStarted","Data":"7a6984b9ac531307f973c7cdf83430f04e6144d2cf172783a012bd7c7cf490bd"} Oct 30 01:04:01.292300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.292049 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:04:01.324925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.324877 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" event={"ID":"2a2e8ebe-01db-458b-ad90-1622880f0629","Type":"ContainerStarted","Data":"629832c4f25f396831a7750c1cc986fb3d311ccbd16eb002cd2b53c426deb75e"} Oct 30 01:04:01.325132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.324955 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" event={"ID":"2a2e8ebe-01db-458b-ad90-1622880f0629","Type":"ContainerStarted","Data":"2083c4bcecbf42883d3e1ff28cf46310febeb81dfdffbce2f6818818a00d69a7"} Oct 30 01:04:01.327230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.325867 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:01.347980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.346732 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" podStartSLOduration=2.346708924 podStartE2EDuration="2.346708924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:01.340696225 +0000 UTC m=+1526.283869418" watchObservedRunningTime="2026-10-30 01:04:01.346708924 +0000 UTC m=+1526.289882117" Oct 30 01:04:01.392625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.392453 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" podStartSLOduration=2.392436547 podStartE2EDuration="2.392436547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:01.387622104 +0000 UTC m=+1526.330795295" watchObservedRunningTime="2026-10-30 01:04:01.392436547 +0000 UTC m=+1526.335609740" Oct 30 01:04:01.561619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.559767 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x"] Oct 30 01:04:01.581969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.581941 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.602693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.602635 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:04:01.619545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.603855 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:04:01.619545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.604469 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:04:01.619545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.604584 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:04:01.619545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.616806 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x"] Oct 30 01:04:01.689231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.689195 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.689419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.689361 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gbq\" (UniqueName: \"kubernetes.io/projected/19020704-51e4-4d08-a073-efcca1c0a35f-kube-api-access-q5gbq\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.689652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.689620 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.791067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.791030 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.791239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.791135 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gbq\" (UniqueName: \"kubernetes.io/projected/19020704-51e4-4d08-a073-efcca1c0a35f-kube-api-access-q5gbq\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.791239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.791190 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.791731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.791701 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.793744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.793701 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.850211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.850130 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gbq\" (UniqueName: \"kubernetes.io/projected/19020704-51e4-4d08-a073-efcca1c0a35f-kube-api-access-q5gbq\") pod \"maas-api-revoke-keys-e2e-api-b-30c405-c299x\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:01.916806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:01.916774 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:02.237132 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:02.237102 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19020704_51e4_4d08_a073_efcca1c0a35f.slice/crio-519c5a5f38704faa090b86d5a25274a2b0827552ad2b25adb6015858541384ad WatchSource:0}: Error finding container 519c5a5f38704faa090b86d5a25274a2b0827552ad2b25adb6015858541384ad: Status 404 returned error can't find the container with id 519c5a5f38704faa090b86d5a25274a2b0827552ad2b25adb6015858541384ad Oct 30 01:04:02.240184 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.240158 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x"] Oct 30 01:04:02.339328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.339288 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" event={"ID":"19020704-51e4-4d08-a073-efcca1c0a35f","Type":"ContainerStarted","Data":"519c5a5f38704faa090b86d5a25274a2b0827552ad2b25adb6015858541384ad"} Oct 30 01:04:02.442285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.442247 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2"] Oct 30 01:04:02.442490 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.442468 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" containerID="cri-o://49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9" gracePeriod=30 Oct 30 01:04:02.449404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.449315 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2"] Oct 30 01:04:02.449799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.449552 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" containerID="cri-o://cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404" gracePeriod=30 Oct 30 01:04:02.460630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.460597 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc"] Oct 30 01:04:02.460849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.460828 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8"] Oct 30 01:04:02.461043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.461013 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" containerID="cri-o://9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149" gracePeriod=30 Oct 30 01:04:02.461656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.461603 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" containerID="cri-o://68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6" gracePeriod=30 Oct 30 01:04:02.774831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.774791 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:04:02.775300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.774839 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:04:02.880205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.880122 4905 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:04:02.880205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:02.880174 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:04:03.358051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:03.357998 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" event={"ID":"19020704-51e4-4d08-a073-efcca1c0a35f","Type":"ContainerStarted","Data":"91126e3e8a2459b3a33208ae176f59b0b0bf0444dbf76b4a177085a0dcb01046"} Oct 30 01:04:03.406813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:03.406758 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" podStartSLOduration=2.406739625 podStartE2EDuration="2.406739625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:03.399351418 +0000 UTC m=+1528.342524610" watchObservedRunningTime="2026-10-30 01:04:03.406739625 +0000 UTC m=+1528.349912822" Oct 30 01:04:04.372935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:04.372899 4905 generic.go:356] "Generic (PLEG): container finished" podID="19020704-51e4-4d08-a073-efcca1c0a35f" containerID="91126e3e8a2459b3a33208ae176f59b0b0bf0444dbf76b4a177085a0dcb01046" exitCode=0 Oct 30 01:04:04.373446 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:04.372945 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" event={"ID":"19020704-51e4-4d08-a073-efcca1c0a35f","Type":"ContainerDied","Data":"91126e3e8a2459b3a33208ae176f59b0b0bf0444dbf76b4a177085a0dcb01046"} Oct 30 01:04:05.030134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.030097 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w"] Oct 30 01:04:05.044302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.044269 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac95ba23-sfd8w"] Oct 30 01:04:05.192929 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.192896 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ac3fef-1c9e-47ae-8a2f-e5e67105596c" path="/var/lib/kubelet/pods/d4ac3fef-1c9e-47ae-8a2f-e5e67105596c/volumes" Oct 30 01:04:05.407096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.407003 4905 generic.go:356] "Generic (PLEG): container finished" podID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerID="9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9" exitCode=0 Oct 30 01:04:05.407096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.407086 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" event={"ID":"e61f6648-27e9-448f-8e07-e2ed4341a36a","Type":"ContainerDied","Data":"9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9"} Oct 30 01:04:05.407660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.407646 4905 scope.go:122] "RemoveContainer" containerID="9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9" Oct 30 01:04:05.409630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.409601 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:05.409765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.409675 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:05.409765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.409710 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:05.722590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.715977 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:05.812389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.812361 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-openshift-ca\") pod \"19020704-51e4-4d08-a073-efcca1c0a35f\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " Oct 30 01:04:05.812616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.812566 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-xks-ca\") pod \"19020704-51e4-4d08-a073-efcca1c0a35f\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " Oct 30 01:04:05.812745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.812673 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19020704-51e4-4d08-a073-efcca1c0a35f-kube-api-access-q5gbq\" (UniqueName: \"kubernetes.io/projected/19020704-51e4-4d08-a073-efcca1c0a35f-kube-api-access-q5gbq\") pod \"19020704-51e4-4d08-a073-efcca1c0a35f\" (UID: \"19020704-51e4-4d08-a073-efcca1c0a35f\") " Oct 30 01:04:05.813476 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.813450 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-openshift-ca" pod "19020704-51e4-4d08-a073-efcca1c0a35f" (UID: "19020704-51e4-4d08-a073-efcca1c0a35f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:05.815871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.815819 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-xks-ca" pod "19020704-51e4-4d08-a073-efcca1c0a35f" (UID: "19020704-51e4-4d08-a073-efcca1c0a35f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:04:05.815871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.815834 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19020704-51e4-4d08-a073-efcca1c0a35f-kube-api-access-q5gbq" pod "19020704-51e4-4d08-a073-efcca1c0a35f" (UID: "19020704-51e4-4d08-a073-efcca1c0a35f"). InnerVolumeSpecName "kube-api-access-q5gbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:05.914765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.914736 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gbq\" (UniqueName: \"kubernetes.io/projected/19020704-51e4-4d08-a073-efcca1c0a35f-kube-api-access-q5gbq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:05.914765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.914762 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:05.914765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:05.914771 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19020704-51e4-4d08-a073-efcca1c0a35f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:06.417076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.417048 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" Oct 30 01:04:06.417076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.417067 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x" event={"ID":"19020704-51e4-4d08-a073-efcca1c0a35f","Type":"ContainerDied","Data":"519c5a5f38704faa090b86d5a25274a2b0827552ad2b25adb6015858541384ad"} Oct 30 01:04:06.417626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.417122 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="519c5a5f38704faa090b86d5a25274a2b0827552ad2b25adb6015858541384ad" Oct 30 01:04:06.419442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.419419 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" event={"ID":"e61f6648-27e9-448f-8e07-e2ed4341a36a","Type":"ContainerStarted","Data":"0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb"} Oct 30 01:04:06.971263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.971232 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w"] Oct 30 01:04:06.972958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.972932 4905 cpu_manager.go:414] "containerMap: removing container" podUID="19020704-51e4-4d08-a073-efcca1c0a35f" containerName="revoke-keys" Oct 30 01:04:06.973145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.973131 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19020704-51e4-4d08-a073-efcca1c0a35f" containerName="revoke-keys" Oct 30 01:04:06.973618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.973585 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="19020704-51e4-4d08-a073-efcca1c0a35f" containerName="revoke-keys" Oct 30 01:04:06.984968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.983128 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w"] Oct 30 01:04:06.984968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:06.983237 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.000599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.000576 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ca20f-openshift-default-dockercfg-z7jsf\"" Oct 30 01:04:07.148563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.123807 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:04:07.148563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.144634 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:04:07.148563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.144689 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:04:07.149610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.149577 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-envoy\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.149840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.149819 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-data\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.149990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.149973 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-podinfo\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.150116 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.150099 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-certs\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.154725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.154695 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-token\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.171547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.154947 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7rb8\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-kube-api-access-c7rb8\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.171986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.171958 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-credential-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.173595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.173569 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.173978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.173957 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istiod-ca-cert\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.302051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.299423 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-envoy\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.302051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.299481 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-data\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.307004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.306967 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-data\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.317017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.299511 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-podinfo\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.317303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.317280 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-certs\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.317509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.317494 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-token\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.317688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.317673 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7rb8\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-kube-api-access-c7rb8\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.317823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.317809 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-credential-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.317949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.317936 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.349543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.318194 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istiod-ca-cert\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.350295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.329201 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-certs\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.351107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.329826 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-credential-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.351247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.330069 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.351348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.333506 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istiod-ca-cert\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.351429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.345048 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-envoy\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.351508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.345627 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-podinfo\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.380540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.363309 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7rb8\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-kube-api-access-c7rb8\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.393595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.384352 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-token\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:07.507812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.507704 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:04:07.608833 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:07.608778 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:08.090411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.090380 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 01:04:08.123563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.123531 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:04:08.123691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.123595 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:04:08.176824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.176723 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w"] Oct 30 01:04:08.183731 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:08.183703 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeca3ce_019e_4b5a_9f23_6dbdc2aceb56.slice/crio-ccb1b93f05a2b073ea2e82d945a95babb8dceaf242835d623ad1cff4d0ca5821 WatchSource:0}: Error finding container ccb1b93f05a2b073ea2e82d945a95babb8dceaf242835d623ad1cff4d0ca5821: Status 404 returned error can't find the container with id ccb1b93f05a2b073ea2e82d945a95babb8dceaf242835d623ad1cff4d0ca5821 Oct 30 01:04:08.188604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.188560 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:08.188720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.188662 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:08.188720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.188708 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203280 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-socket\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203376 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-podinfo\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203458 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-envoy\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203537 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-credential-socket\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203577 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istiod-ca-cert\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203634 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-token\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203684 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-kube-api-access-m9nnh\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-kube-api-access-m9nnh\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203788 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-data\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.203922 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-certs\") pod \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\" (UID: \"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b\") " Oct 30 01:04:08.204834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.204778 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-socket" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.205020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.204996 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-credential-socket" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.205139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.205075 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-data" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.205419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.205397 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-certs" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.205602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.205584 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istiod-ca-cert" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:08.205740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.205724 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.205846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.205831 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.205923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.205914 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.208749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.208720 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-kube-api-access-m9nnh" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "kube-api-access-m9nnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.209138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.209071 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-token" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.209432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.209369 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-podinfo" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:08.211975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.211855 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-envoy" pod "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" (UID: "b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.308650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.308622 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9nnh\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-kube-api-access-m9nnh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.308828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.308812 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.308938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.308926 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.309031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.309017 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.309126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.309115 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.309215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.309205 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.317866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.317842 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 01:04:08.408405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.408375 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 01:04:08.409883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.409857 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-data\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.409969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.409944 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-socket\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.409999 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-envoy\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410095 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-credential-socket\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410155 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-kube-api-access-bzssm\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-kube-api-access-bzssm\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410297 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80abb66d-7e8c-4524-8902-e4afd8221305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80abb66d-7e8c-4524-8902-e4afd8221305-istio-podinfo\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410330 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-istio-token\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-istio-token\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410365 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-certs\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410417 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80abb66d-7e8c-4524-8902-e4afd8221305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80abb66d-7e8c-4524-8902-e4afd8221305-istiod-ca-cert\") pod \"80abb66d-7e8c-4524-8902-e4afd8221305\" (UID: \"80abb66d-7e8c-4524-8902-e4afd8221305\") " Oct 30 01:04:08.410801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410778 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-credential-socket" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.410970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.410953 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-data" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.411118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.411103 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-socket" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.411293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.411277 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 01:04:08.411723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.411700 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.411812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.411729 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.411812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.411743 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.412257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.412232 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80abb66d-7e8c-4524-8902-e4afd8221305-istiod-ca-cert" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:08.413375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.413343 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80abb66d-7e8c-4524-8902-e4afd8221305-istio-podinfo" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:08.413928 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.413884 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-certs" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.415161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.415134 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-envoy" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.474763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.474732 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-istio-token" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.493605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.493573 4905 generic.go:356] "Generic (PLEG): container finished" podID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerID="68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6" exitCode=0 Oct 30 01:04:08.493754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.493603 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" event={"ID":"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6","Type":"ContainerDied","Data":"68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6"} Oct 30 01:04:08.493754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.493652 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" event={"ID":"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6","Type":"ContainerDied","Data":"8e822b5e1401c1f20fe4659d1c245a7ddfe3fe978e4ea33decaea8632d8ac236"} Oct 30 01:04:08.493754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.493678 4905 scope.go:122] "RemoveContainer" containerID="68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6" Oct 30 01:04:08.493754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.493585 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc" Oct 30 01:04:08.493754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.493715 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-kube-api-access-bzssm" pod "80abb66d-7e8c-4524-8902-e4afd8221305" (UID: "80abb66d-7e8c-4524-8902-e4afd8221305"). InnerVolumeSpecName "kube-api-access-bzssm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513062 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-certs\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513118 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-token\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-token\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513156 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istiod-ca-cert\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513191 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-socket\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513253 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-token\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-token\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513299 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-data\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513369 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-envoy\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513396 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-credential-socket\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513460 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istiod-ca-cert\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513492 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-credential-socket\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513544 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-podinfo\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513595 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-data\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513685 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-envoy\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513728 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-podinfo\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513768 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-kube-api-access-vvf4l\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-kube-api-access-vvf4l\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513865 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-kube-api-access-jnqpg\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-kube-api-access-jnqpg\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513909 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-certs\") pod \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\" (UID: \"fdf24b36-3aa9-4cf8-b548-2d3a58539bf6\") " Oct 30 01:04:08.514158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.513971 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-socket\") pod \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\" (UID: \"feaf92d1-706a-4379-b59b-adf69e1fdaf5\") " Oct 30 01:04:08.518224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.515711 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-credential-socket" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.522735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.521585 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-credential-socket" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.522735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.522230 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-certs" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.541448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.541400 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istiod-ca-cert" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:08.560429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.559916 4905 generic.go:356] "Generic (PLEG): container finished" podID="80abb66d-7e8c-4524-8902-e4afd8221305" containerID="cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404" exitCode=0 Oct 30 01:04:08.560429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.560049 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" event={"ID":"80abb66d-7e8c-4524-8902-e4afd8221305","Type":"ContainerDied","Data":"cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404"} Oct 30 01:04:08.560429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.560098 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" event={"ID":"80abb66d-7e8c-4524-8902-e4afd8221305","Type":"ContainerDied","Data":"3354cd9a48775cbfe4a1b36eaf5b142b515da913776bb5cbcc66049c63303ba6"} Oct 30 01:04:08.560429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.560174 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2" Oct 30 01:04:08.590033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.585701 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-socket" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.590033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.585910 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-token" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.590033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.586017 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istiod-ca-cert" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:08.590033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.586242 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-socket" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.595254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.592365 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-podinfo" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:08.595254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.592761 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-kube-api-access-jnqpg" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "kube-api-access-jnqpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.595254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.592841 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-certs" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.595254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.594472 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-data" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.599048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.595792 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-data" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.599048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.598808 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-token" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.599666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.599634 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.599814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.599800 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.599899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.599889 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzssm\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-kube-api-access-bzssm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.599976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.599966 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.600063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600052 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.600144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600135 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.600221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600212 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80abb66d-7e8c-4524-8902-e4afd8221305-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.600292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600283 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80abb66d-7e8c-4524-8902-e4afd8221305-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.600367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600339 4905 generic.go:356] "Generic (PLEG): container finished" podID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerID="9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149" exitCode=0 Oct 30 01:04:08.600503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600491 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" Oct 30 01:04:08.600784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600498 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" event={"ID":"feaf92d1-706a-4379-b59b-adf69e1fdaf5","Type":"ContainerDied","Data":"9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149"} Oct 30 01:04:08.600902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600888 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8" event={"ID":"feaf92d1-706a-4379-b59b-adf69e1fdaf5","Type":"ContainerDied","Data":"ccf01b961bfad1f63a419fad16912de5b144e1d30e438dfdc6a6baab4094c34b"} Oct 30 01:04:08.600994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.600351 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80abb66d-7e8c-4524-8902-e4afd8221305-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.601088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.601077 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80abb66d-7e8c-4524-8902-e4afd8221305-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.603790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.603759 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" event={"ID":"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56","Type":"ContainerStarted","Data":"2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1"} Oct 30 01:04:08.603949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.603814 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" event={"ID":"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56","Type":"ContainerStarted","Data":"ccb1b93f05a2b073ea2e82d945a95babb8dceaf242835d623ad1cff4d0ca5821"} Oct 30 01:04:08.613156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.611592 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:08.613156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.611684 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-podinfo" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:08.613156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.612795 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-envoy" pod "feaf92d1-706a-4379-b59b-adf69e1fdaf5" (UID: "feaf92d1-706a-4379-b59b-adf69e1fdaf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.613156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.612887 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-kube-api-access-vvf4l" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "kube-api-access-vvf4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:08.614040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.614011 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-envoy" pod "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" (UID: "fdf24b36-3aa9-4cf8-b548-2d3a58539bf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:08.616335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.615495 4905 scope.go:122] "RemoveContainer" containerID="68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6" Oct 30 01:04:08.616335 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:08.616036 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6\": container with ID starting with 68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6 not found: ID does not exist" containerID="68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6" Oct 30 01:04:08.616335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.616062 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6"} err="failed to get container status \"68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6\": rpc error: code = NotFound desc = could not find container \"68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6\": container with ID starting with 68329141dca0e408d2a05ac95b0e442603d0d4f0e8d9a197670b508d230f52d6 not found: ID does not exist" Oct 30 01:04:08.616335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.616078 4905 scope.go:122] "RemoveContainer" containerID="cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404" Oct 30 01:04:08.667547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.624781 4905 generic.go:356] "Generic (PLEG): container finished" podID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerID="49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9" exitCode=0 Oct 30 01:04:08.667547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.625846 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" Oct 30 01:04:08.667547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.645170 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" event={"ID":"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b","Type":"ContainerDied","Data":"49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9"} Oct 30 01:04:08.667547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.645232 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2" event={"ID":"b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b","Type":"ContainerDied","Data":"f2519c38f9d1495d3626e11b87b7f5ca336701e2a1697fb37df0c7cd1f575001"} Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705282 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705318 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705332 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvf4l\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-kube-api-access-vvf4l\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705346 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnqpg\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-kube-api-access-jnqpg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705361 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705376 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705389 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705402 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705415 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705430 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705443 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feaf92d1-706a-4379-b59b-adf69e1fdaf5-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705455 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705467 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.705779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.705478 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:08.712971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.712898 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podStartSLOduration=2.712880028 podStartE2EDuration="2.712880028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:08.699259351 +0000 UTC m=+1533.642432579" watchObservedRunningTime="2026-10-30 01:04:08.712880028 +0000 UTC m=+1533.656053220" Oct 30 01:04:08.779358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.777823 4905 scope.go:122] "RemoveContainer" containerID="cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404" Oct 30 01:04:08.780106 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:08.779976 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404\": container with ID starting with cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404 not found: ID does not exist" containerID="cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404" Oct 30 01:04:08.780106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.780008 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404"} err="failed to get container status \"cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404\": rpc error: code = NotFound desc = could not find container \"cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404\": container with ID starting with cec0f16c5c8765594f75698e2a864e5726579361c573edb811b62fc08d30c404 not found: ID does not exist" Oct 30 01:04:08.780106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.780030 4905 scope.go:122] "RemoveContainer" containerID="9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149" Oct 30 01:04:08.803666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.802674 4905 scope.go:122] "RemoveContainer" containerID="f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312" Oct 30 01:04:08.901593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:08.893161 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2"] Oct 30 01:04:08.957056 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:08.957015 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80abb66d_7e8c_4524_8902_e4afd8221305.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80abb66d_7e8c_4524_8902_e4afd8221305.slice/crio-3354cd9a48775cbfe4a1b36eaf5b142b515da913776bb5cbcc66049c63303ba6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf24b36_3aa9_4cf8_b548_2d3a58539bf6.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:04:09.007795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.007753 4905 scope.go:122] "RemoveContainer" containerID="9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149" Oct 30 01:04:09.008157 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:09.008124 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149\": container with ID starting with 9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149 not found: ID does not exist" containerID="9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149" Oct 30 01:04:09.008261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.008163 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149"} err="failed to get container status \"9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149\": rpc error: code = NotFound desc = could not find container \"9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149\": container with ID starting with 9225b1297c95179dbe60b7ccf5d986a6fba0897eb2cb02841a06e306271cf149 not found: ID does not exist" Oct 30 01:04:09.008261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.008185 4905 scope.go:122] "RemoveContainer" containerID="f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312" Oct 30 01:04:09.008495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.008479 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-cght2"] Oct 30 01:04:09.008955 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:09.008933 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312\": container with ID starting with f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312 not found: ID does not exist" containerID="f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312" Oct 30 01:04:09.009051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.008962 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312"} err="failed to get container status \"f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312\": rpc error: code = NotFound desc = could not find container \"f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312\": container with ID starting with f8093210c610fd2262e6fc0e599d7f646baf9f9d7098598cca99f940ca331312 not found: ID does not exist" Oct 30 01:04:09.009051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.008984 4905 scope.go:122] "RemoveContainer" containerID="49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9" Oct 30 01:04:09.022051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.021985 4905 scope.go:122] "RemoveContainer" containerID="49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9" Oct 30 01:04:09.022253 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:09.022228 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9\": container with ID starting with 49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9 not found: ID does not exist" containerID="49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9" Oct 30 01:04:09.022341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.022258 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9"} err="failed to get container status \"49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9\": rpc error: code = NotFound desc = could not find container \"49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9\": container with ID starting with 49399975fbb58b2cae6f7aa5ee0e3aaa28af759c63ba5bf589b37493c22169c9 not found: ID does not exist" Oct 30 01:04:09.054109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.054080 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2"] Oct 30 01:04:09.072075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.072041 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-6m9m2"] Oct 30 01:04:09.092197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.091744 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc"] Oct 30 01:04:09.095612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.095592 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-2mvtc"] Oct 30 01:04:09.121841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.121803 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:04:09.121970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.121864 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:04:09.168257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.168141 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8"] Oct 30 01:04:09.174462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.174434 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-qj5c8"] Oct 30 01:04:09.192220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.192195 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" path="/var/lib/kubelet/pods/80abb66d-7e8c-4524-8902-e4afd8221305/volumes" Oct 30 01:04:09.192718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.192703 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" path="/var/lib/kubelet/pods/b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b/volumes" Oct 30 01:04:09.193186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.193173 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" path="/var/lib/kubelet/pods/fdf24b36-3aa9-4cf8-b548-2d3a58539bf6/volumes" Oct 30 01:04:09.193972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.193958 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" path="/var/lib/kubelet/pods/feaf92d1-706a-4379-b59b-adf69e1fdaf5/volumes" Oct 30 01:04:09.615916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.615883 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:09.677177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.677140 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:09.678212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:09.678192 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:04:10.122013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:10.121982 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:04:10.122236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:10.122034 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:04:11.121664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:11.121628 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:04:11.122039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:11.121689 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:04:12.062778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:12.062742 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9"] Oct 30 01:04:12.068391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:12.068357 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0997bc-7vkl9"] Oct 30 01:04:12.122325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:12.122296 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:04:12.122735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:12.122364 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:04:12.162531 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:12.162480 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh"] Oct 30 01:04:12.182171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:12.182134 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6"] Oct 30 01:04:12.418498 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:12.418402 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:04:13.237545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:13.237145 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f530630-85af-484a-9451-d9d82052f994" path="/var/lib/kubelet/pods/9f530630-85af-484a-9451-d9d82052f994/volumes" Oct 30 01:04:13.297667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:13.297636 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:04:13.299337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:13.299308 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:04:13.362164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:13.362134 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:04:13.363627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:13.363602 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:04:14.133626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.133595 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9"] Oct 30 01:04:14.134287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134274 4905 cpu_manager.go:414] "containerMap: removing container" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" Oct 30 01:04:14.134325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134289 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" Oct 30 01:04:14.134380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134328 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" Oct 30 01:04:14.134380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134341 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" Oct 30 01:04:14.134380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134358 4905 cpu_manager.go:414] "containerMap: removing container" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" Oct 30 01:04:14.134380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134363 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" Oct 30 01:04:14.134380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134373 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" Oct 30 01:04:14.134380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134378 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" Oct 30 01:04:14.134603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134476 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" Oct 30 01:04:14.134603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134490 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf24b36-3aa9-4cf8-b548-2d3a58539bf6" containerName="istio-proxy" Oct 30 01:04:14.134603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134498 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="80abb66d-7e8c-4524-8902-e4afd8221305" containerName="istio-proxy" Oct 30 01:04:14.134603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.134504 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a0ddfc-9234-49cf-ae53-41b9cd22ee7b" containerName="istio-proxy" Oct 30 01:04:14.139866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.139846 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.158028 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.158002 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2ca20f-dockercfg-xggjz\"" Oct 30 01:04:14.166671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.166645 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9"] Oct 30 01:04:14.170014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.169992 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2ca20f\"" Oct 30 01:04:14.251544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.233778 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cc2\" (UniqueName: \"kubernetes.io/projected/06211214-57e4-432e-b254-0b30fea59733-kube-api-access-j9cc2\") pod \"payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.251544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.234022 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06211214-57e4-432e-b254-0b30fea59733-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.305544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.293775 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25"] Oct 30 01:04:14.305544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.294987 4905 cpu_manager.go:414] "containerMap: removing container" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" Oct 30 01:04:14.305544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.295011 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" Oct 30 01:04:14.305544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.295202 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="feaf92d1-706a-4379-b59b-adf69e1fdaf5" containerName="istio-proxy" Oct 30 01:04:14.370807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.348832 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06211214-57e4-432e-b254-0b30fea59733-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.370807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.349099 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cc2\" (UniqueName: \"kubernetes.io/projected/06211214-57e4-432e-b254-0b30fea59733-kube-api-access-j9cc2\") pod \"payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.386495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.376924 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:14.387561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.386896 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06211214-57e4-432e-b254-0b30fea59733-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.468441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.468043 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rj7x\" (UniqueName: \"kubernetes.io/projected/7c94fbff-b58f-4434-a509-01f353a46658-kube-api-access-2rj7x\") pod \"payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:14.468441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.468101 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94fbff-b58f-4434-a509-01f353a46658-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:14.521630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.521580 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25"] Oct 30 01:04:14.573590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.570415 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rj7x\" (UniqueName: \"kubernetes.io/projected/7c94fbff-b58f-4434-a509-01f353a46658-kube-api-access-2rj7x\") pod \"payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:14.573590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.570480 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94fbff-b58f-4434-a509-01f353a46658-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:14.573590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.572097 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94fbff-b58f-4434-a509-01f353a46658-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:14.578458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.575765 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cc2\" (UniqueName: \"kubernetes.io/projected/06211214-57e4-432e-b254-0b30fea59733-kube-api-access-j9cc2\") pod \"payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.611502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.611469 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rj7x\" (UniqueName: \"kubernetes.io/projected/7c94fbff-b58f-4434-a509-01f353a46658-kube-api-access-2rj7x\") pod \"payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:14.767138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.767102 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:14.853035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:14.852765 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:15.079364 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:15.079337 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06211214_57e4_432e_b254_0b30fea59733.slice/crio-6bd257c9479f4cb00b977ed2dedcf27320f4e14fa3499832a45443c220ffff36 WatchSource:0}: Error finding container 6bd257c9479f4cb00b977ed2dedcf27320f4e14fa3499832a45443c220ffff36: Status 404 returned error can't find the container with id 6bd257c9479f4cb00b977ed2dedcf27320f4e14fa3499832a45443c220ffff36 Oct 30 01:04:15.080947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:15.080916 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9"] Oct 30 01:04:15.176585 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:15.176561 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c94fbff_b58f_4434_a509_01f353a46658.slice/crio-204ef9699a6b5f15bb47883a6846c69090d42cc823bb88042fd10dab1a843263 WatchSource:0}: Error finding container 204ef9699a6b5f15bb47883a6846c69090d42cc823bb88042fd10dab1a843263: Status 404 returned error can't find the container with id 204ef9699a6b5f15bb47883a6846c69090d42cc823bb88042fd10dab1a843263 Oct 30 01:04:15.178428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:15.178405 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25"] Oct 30 01:04:16.021805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.021766 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" event={"ID":"06211214-57e4-432e-b254-0b30fea59733","Type":"ContainerStarted","Data":"30b65753688a50a8b29d5cd1b39999911ac0cd0d0dd6f4910d88b88bf492f99d"} Oct 30 01:04:16.022404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.022378 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" event={"ID":"06211214-57e4-432e-b254-0b30fea59733","Type":"ContainerStarted","Data":"6bd257c9479f4cb00b977ed2dedcf27320f4e14fa3499832a45443c220ffff36"} Oct 30 01:04:16.023850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.023828 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:16.048559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.048505 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" event={"ID":"7c94fbff-b58f-4434-a509-01f353a46658","Type":"ContainerStarted","Data":"ad9362b620589eb61f4eea97305ee9cdaee5820e3f8ee1de626616ab1e0bb9a6"} Oct 30 01:04:16.048802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.048783 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" event={"ID":"7c94fbff-b58f-4434-a509-01f353a46658","Type":"ContainerStarted","Data":"204ef9699a6b5f15bb47883a6846c69090d42cc823bb88042fd10dab1a843263"} Oct 30 01:04:16.049213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.049195 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:16.110819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.108962 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" podStartSLOduration=2.108908779 podStartE2EDuration="2.108908779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:16.093336039 +0000 UTC m=+1541.036509232" watchObservedRunningTime="2026-10-30 01:04:16.108908779 +0000 UTC m=+1541.052081969" Oct 30 01:04:16.150547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.134702 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" podStartSLOduration=2.134683536 podStartE2EDuration="2.134683536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:16.133982696 +0000 UTC m=+1541.077155902" watchObservedRunningTime="2026-10-30 01:04:16.134683536 +0000 UTC m=+1541.077856730" Oct 30 01:04:16.648575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.645637 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km"] Oct 30 01:04:16.669706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:16.669671 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b814fb-gw4km"] Oct 30 01:04:17.163849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.163806 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" podUID="f4641871-a640-4404-adec-c62f1a46acb4" containerName="payload-processing" containerID="cri-o://50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39" gracePeriod=25 Oct 30 01:04:17.182552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.182494 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" podUID="1330a7f0-0608-435a-b516-3a6be58d692b" containerName="payload-pre-processing" containerID="cri-o://3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae" gracePeriod=25 Oct 30 01:04:17.196218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.196182 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc88044-2936-45ab-a3a1-9bb59e3267a8" path="/var/lib/kubelet/pods/fdc88044-2936-45ab-a3a1-9bb59e3267a8/volumes" Oct 30 01:04:17.558551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.554870 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:04:17.595787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.595762 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:04:17.612635 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.612607 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4641871-a640-4404-adec-c62f1a46acb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4641871-a640-4404-adec-c62f1a46acb4-plugins-config-volume\") pod \"f4641871-a640-4404-adec-c62f1a46acb4\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " Oct 30 01:04:17.613000 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.612974 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4641871-a640-4404-adec-c62f1a46acb4-plugins-config-volume" pod "f4641871-a640-4404-adec-c62f1a46acb4" (UID: "f4641871-a640-4404-adec-c62f1a46acb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:17.613192 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.613174 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4641871-a640-4404-adec-c62f1a46acb4-kube-api-access-5cqmv\" (UniqueName: \"kubernetes.io/projected/f4641871-a640-4404-adec-c62f1a46acb4-kube-api-access-5cqmv\") pod \"f4641871-a640-4404-adec-c62f1a46acb4\" (UID: \"f4641871-a640-4404-adec-c62f1a46acb4\") " Oct 30 01:04:17.613920 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.613896 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4641871-a640-4404-adec-c62f1a46acb4-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:17.615343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.615320 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4641871-a640-4404-adec-c62f1a46acb4-kube-api-access-5cqmv" pod "f4641871-a640-4404-adec-c62f1a46acb4" (UID: "f4641871-a640-4404-adec-c62f1a46acb4"). InnerVolumeSpecName "kube-api-access-5cqmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:17.714894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.714814 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1330a7f0-0608-435a-b516-3a6be58d692b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1330a7f0-0608-435a-b516-3a6be58d692b-plugins-config-volume\") pod \"1330a7f0-0608-435a-b516-3a6be58d692b\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " Oct 30 01:04:17.715057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.714958 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1330a7f0-0608-435a-b516-3a6be58d692b-kube-api-access-kqgst\" (UniqueName: \"kubernetes.io/projected/1330a7f0-0608-435a-b516-3a6be58d692b-kube-api-access-kqgst\") pod \"1330a7f0-0608-435a-b516-3a6be58d692b\" (UID: \"1330a7f0-0608-435a-b516-3a6be58d692b\") " Oct 30 01:04:17.715188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.715165 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1330a7f0-0608-435a-b516-3a6be58d692b-plugins-config-volume" pod "1330a7f0-0608-435a-b516-3a6be58d692b" (UID: "1330a7f0-0608-435a-b516-3a6be58d692b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:17.715452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.715437 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1330a7f0-0608-435a-b516-3a6be58d692b-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:17.715503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.715459 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cqmv\" (UniqueName: \"kubernetes.io/projected/f4641871-a640-4404-adec-c62f1a46acb4-kube-api-access-5cqmv\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:17.717234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.717209 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1330a7f0-0608-435a-b516-3a6be58d692b-kube-api-access-kqgst" pod "1330a7f0-0608-435a-b516-3a6be58d692b" (UID: "1330a7f0-0608-435a-b516-3a6be58d692b"). InnerVolumeSpecName "kube-api-access-kqgst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:17.816427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:17.816394 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqgst\" (UniqueName: \"kubernetes.io/projected/1330a7f0-0608-435a-b516-3a6be58d692b-kube-api-access-kqgst\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:18.153551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.152569 4905 generic.go:356] "Generic (PLEG): container finished" podID="1330a7f0-0608-435a-b516-3a6be58d692b" containerID="3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae" exitCode=0 Oct 30 01:04:18.153551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.152877 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" event={"ID":"1330a7f0-0608-435a-b516-3a6be58d692b","Type":"ContainerDied","Data":"3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae"} Oct 30 01:04:18.153551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.152951 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" event={"ID":"1330a7f0-0608-435a-b516-3a6be58d692b","Type":"ContainerDied","Data":"4e9769fb959b47f30f12704784392c4f6135fd54971a2e2f21f0f70bd60ba647"} Oct 30 01:04:18.153551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.152983 4905 scope.go:122] "RemoveContainer" containerID="3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae" Oct 30 01:04:18.153551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.153144 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6" Oct 30 01:04:18.264022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.262568 4905 generic.go:356] "Generic (PLEG): container finished" podID="f4641871-a640-4404-adec-c62f1a46acb4" containerID="50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39" exitCode=0 Oct 30 01:04:18.264022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.263586 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" Oct 30 01:04:18.300812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.273663 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" event={"ID":"f4641871-a640-4404-adec-c62f1a46acb4","Type":"ContainerDied","Data":"50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39"} Oct 30 01:04:18.300812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.273735 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh" event={"ID":"f4641871-a640-4404-adec-c62f1a46acb4","Type":"ContainerDied","Data":"98228d2fb0be75ee6a926025104060bcfed378675041c3a2c322f3dff572561c"} Oct 30 01:04:18.300812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.300723 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6"] Oct 30 01:04:18.321356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.316333 4905 scope.go:122] "RemoveContainer" containerID="3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae" Oct 30 01:04:18.321356 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:18.316795 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae\": container with ID starting with 3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae not found: ID does not exist" containerID="3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae" Oct 30 01:04:18.321356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.316833 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae"} err="failed to get container status \"3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae\": rpc error: code = NotFound desc = could not find container \"3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae\": container with ID starting with 3221aea3ae09c3944d2ede43f4242baaf99e172c3cee3c388a4deb856f43beae not found: ID does not exist" Oct 30 01:04:18.321356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.316854 4905 scope.go:122] "RemoveContainer" containerID="50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39" Oct 30 01:04:18.321356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.318418 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30c405-5f8794dd8c-5xpk6"] Oct 30 01:04:18.337267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.337231 4905 scope.go:122] "RemoveContainer" containerID="50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39" Oct 30 01:04:18.337844 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:18.337821 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39\": container with ID starting with 50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39 not found: ID does not exist" containerID="50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39" Oct 30 01:04:18.338016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.337991 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39"} err="failed to get container status \"50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39\": rpc error: code = NotFound desc = could not find container \"50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39\": container with ID starting with 50d33582246f753c467afe429c7e78765091639f3de0548a0ecb2f99e3e5cc39 not found: ID does not exist" Oct 30 01:04:18.368350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.368315 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh"] Oct 30 01:04:18.372303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:18.372271 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30c405-578d7b5474-lq4jh"] Oct 30 01:04:19.192408 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:19.192369 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1330a7f0-0608-435a-b516-3a6be58d692b" path="/var/lib/kubelet/pods/1330a7f0-0608-435a-b516-3a6be58d692b/volumes" Oct 30 01:04:19.192933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:19.192907 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4641871-a640-4404-adec-c62f1a46acb4" path="/var/lib/kubelet/pods/f4641871-a640-4404-adec-c62f1a46acb4/volumes" Oct 30 01:04:21.669015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.668983 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n"] Oct 30 01:04:21.669917 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.669891 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1330a7f0-0608-435a-b516-3a6be58d692b" containerName="payload-pre-processing" Oct 30 01:04:21.669917 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.669917 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1330a7f0-0608-435a-b516-3a6be58d692b" containerName="payload-pre-processing" Oct 30 01:04:21.670073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.670009 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f4641871-a640-4404-adec-c62f1a46acb4" containerName="payload-processing" Oct 30 01:04:21.670073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.670018 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4641871-a640-4404-adec-c62f1a46acb4" containerName="payload-processing" Oct 30 01:04:21.670196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.670183 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4641871-a640-4404-adec-c62f1a46acb4" containerName="payload-processing" Oct 30 01:04:21.670236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.670201 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1330a7f0-0608-435a-b516-3a6be58d692b" containerName="payload-pre-processing" Oct 30 01:04:21.676657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.676629 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.701330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.696749 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-bdf6f3-openshift-default-dockercfg-82m2q\"" Oct 30 01:04:21.741577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.741541 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n"] Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.831842 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-envoy\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.831910 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-credential-socket\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.831973 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-certs\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.831996 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-token\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.832112 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-podinfo\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.832147 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-data\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.832181 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b272f295-7d0a-49d2-af8d-4970059d17a7-istiod-ca-cert\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.832268 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvktb\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-kube-api-access-rvktb\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.844544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.832471 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-socket\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.939919 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvktb\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-kube-api-access-rvktb\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940085 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-socket\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940141 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-envoy\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940178 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-credential-socket\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940230 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-certs\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940256 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-token\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940359 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-podinfo\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940393 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-data\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.940426 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b272f295-7d0a-49d2-af8d-4970059d17a7-istiod-ca-cert\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.942550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.941227 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b272f295-7d0a-49d2-af8d-4970059d17a7-istiod-ca-cert\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.954544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.950037 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-socket\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.954544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.950282 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-data\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.954544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.950536 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-credential-socket\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:21.960579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:21.955559 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-certs\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:22.063587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.061005 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-token\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:22.063587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.063140 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-envoy\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:22.068497 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.068112 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-podinfo\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:22.125543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.125481 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvktb\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-kube-api-access-rvktb\") pod \"e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:22.347870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.347770 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:22.746862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.746835 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n"] Oct 30 01:04:22.747218 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:22.747112 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb272f295_7d0a_49d2_af8d_4970059d17a7.slice/crio-1a0712635b209165648c99a6d3826d3452e0a32e1a7ccce59658c70e09de33a3 WatchSource:0}: Error finding container 1a0712635b209165648c99a6d3826d3452e0a32e1a7ccce59658c70e09de33a3: Status 404 returned error can't find the container with id 1a0712635b209165648c99a6d3826d3452e0a32e1a7ccce59658c70e09de33a3 Oct 30 01:04:22.751134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.750888 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:22.751134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.750975 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:22.751134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:22.751018 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:23.315154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.315115 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" event={"ID":"b272f295-7d0a-49d2-af8d-4970059d17a7","Type":"ContainerStarted","Data":"0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25"} Oct 30 01:04:23.315341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.315169 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" event={"ID":"b272f295-7d0a-49d2-af8d-4970059d17a7","Type":"ContainerStarted","Data":"1a0712635b209165648c99a6d3826d3452e0a32e1a7ccce59658c70e09de33a3"} Oct 30 01:04:23.348888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.348849 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:23.356380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.356347 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:23.380952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.380898 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podStartSLOduration=2.380878239 podStartE2EDuration="2.380878239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:23.373344649 +0000 UTC m=+1548.316517843" watchObservedRunningTime="2026-10-30 01:04:23.380878239 +0000 UTC m=+1548.324051490" Oct 30 01:04:23.692653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.686392 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p"] Oct 30 01:04:23.692653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.686661 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" containerID="cri-o://27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2" gracePeriod=30 Oct 30 01:04:23.695482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.695244 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw"] Oct 30 01:04:23.695871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:23.695695 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" containerID="cri-o://c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6" gracePeriod=30 Oct 30 01:04:24.246482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.246440 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr"] Oct 30 01:04:24.251951 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.251923 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.269785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.269759 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr"] Oct 30 01:04:24.307689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.307661 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:04:24.307839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.307698 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:04:24.322652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.322604 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.322816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.322730 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bxx\" (UniqueName: \"kubernetes.io/projected/338b099e-30c4-4f31-a6c8-8be499d56267-kube-api-access-s9bxx\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.322934 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.322913 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.325538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.325106 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:24.326476 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.326456 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:04:24.351396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.351348 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:04:24.351396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.351358 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:04:24.426648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.424641 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.426648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.424714 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bxx\" (UniqueName: \"kubernetes.io/projected/338b099e-30c4-4f31-a6c8-8be499d56267-kube-api-access-s9bxx\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.426648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.424904 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.426648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.425884 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.431575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.428966 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.447956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.447927 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bxx\" (UniqueName: \"kubernetes.io/projected/338b099e-30c4-4f31-a6c8-8be499d56267-kube-api-access-s9bxx\") pod \"maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.578481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.578175 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:24.897624 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:24.897584 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338b099e_30c4_4f31_a6c8_8be499d56267.slice/crio-7890e85ef7cbb3f35bf0fadb62ff5c4a04735866aa1bf7e33fa699f8fb770234 WatchSource:0}: Error finding container 7890e85ef7cbb3f35bf0fadb62ff5c4a04735866aa1bf7e33fa699f8fb770234: Status 404 returned error can't find the container with id 7890e85ef7cbb3f35bf0fadb62ff5c4a04735866aa1bf7e33fa699f8fb770234 Oct 30 01:04:24.898581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:24.898553 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr"] Oct 30 01:04:25.342882 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:25.342851 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" event={"ID":"338b099e-30c4-4f31-a6c8-8be499d56267","Type":"ContainerStarted","Data":"7890e85ef7cbb3f35bf0fadb62ff5c4a04735866aa1bf7e33fa699f8fb770234"} Oct 30 01:04:26.431597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:26.426462 4905 generic.go:356] "Generic (PLEG): container finished" podID="338b099e-30c4-4f31-a6c8-8be499d56267" containerID="8995d0e053ebc50138564adad20bdde942c2404a9f5f69a636b186687e926124" exitCode=0 Oct 30 01:04:26.431597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:26.427924 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" event={"ID":"338b099e-30c4-4f31-a6c8-8be499d56267","Type":"ContainerDied","Data":"8995d0e053ebc50138564adad20bdde942c2404a9f5f69a636b186687e926124"} Oct 30 01:04:27.067532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.067432 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:04:27.440453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.440371 4905 generic.go:356] "Generic (PLEG): container finished" podID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerID="9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a" exitCode=0 Oct 30 01:04:27.441010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.440444 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" event={"ID":"9370e3cd-a5c6-4b32-833b-68195297a32f","Type":"ContainerDied","Data":"9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a"} Oct 30 01:04:27.441010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.440918 4905 scope.go:122] "RemoveContainer" containerID="9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a" Oct 30 01:04:27.443459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.443421 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:27.443645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.443532 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:27.443645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.443586 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:27.694131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.694054 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:04:27.694131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.694110 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:04:27.709855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.709820 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:27.814714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.814675 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-xks-ca\") pod \"338b099e-30c4-4f31-a6c8-8be499d56267\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " Oct 30 01:04:27.814906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.814726 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338b099e-30c4-4f31-a6c8-8be499d56267-kube-api-access-s9bxx\" (UniqueName: \"kubernetes.io/projected/338b099e-30c4-4f31-a6c8-8be499d56267-kube-api-access-s9bxx\") pod \"338b099e-30c4-4f31-a6c8-8be499d56267\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " Oct 30 01:04:27.814906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.814831 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-openshift-ca\") pod \"338b099e-30c4-4f31-a6c8-8be499d56267\" (UID: \"338b099e-30c4-4f31-a6c8-8be499d56267\") " Oct 30 01:04:27.816011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.815984 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-openshift-ca" pod "338b099e-30c4-4f31-a6c8-8be499d56267" (UID: "338b099e-30c4-4f31-a6c8-8be499d56267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:27.818855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.818828 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-xks-ca" pod "338b099e-30c4-4f31-a6c8-8be499d56267" (UID: "338b099e-30c4-4f31-a6c8-8be499d56267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:04:27.822685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.822661 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338b099e-30c4-4f31-a6c8-8be499d56267-kube-api-access-s9bxx" pod "338b099e-30c4-4f31-a6c8-8be499d56267" (UID: "338b099e-30c4-4f31-a6c8-8be499d56267"). InnerVolumeSpecName "kube-api-access-s9bxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:27.916711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.916684 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:27.916711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.916708 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9bxx\" (UniqueName: \"kubernetes.io/projected/338b099e-30c4-4f31-a6c8-8be499d56267-kube-api-access-s9bxx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:27.916711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:27.916719 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338b099e-30c4-4f31-a6c8-8be499d56267-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:28.044090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.044053 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq"] Oct 30 01:04:28.044944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.044925 4905 cpu_manager.go:414] "containerMap: removing container" podUID="338b099e-30c4-4f31-a6c8-8be499d56267" containerName="revoke-keys" Oct 30 01:04:28.044944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.044943 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338b099e-30c4-4f31-a6c8-8be499d56267" containerName="revoke-keys" Oct 30 01:04:28.045129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.045087 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="338b099e-30c4-4f31-a6c8-8be499d56267" containerName="revoke-keys" Oct 30 01:04:28.048894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.048874 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.065689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.065560 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-bdf6f3-dockercfg-gltt9\"" Oct 30 01:04:28.065689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.065663 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-bdf6f3\"" Oct 30 01:04:28.072537 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.071656 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq"] Oct 30 01:04:28.142744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.142699 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsjp\" (UniqueName: \"kubernetes.io/projected/a469becb-c0d0-40cc-b426-3287026d49aa-kube-api-access-dlsjp\") pod \"payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.142962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.142850 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a469becb-c0d0-40cc-b426-3287026d49aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.250544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.249312 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a469becb-c0d0-40cc-b426-3287026d49aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.250544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.249600 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsjp\" (UniqueName: \"kubernetes.io/projected/a469becb-c0d0-40cc-b426-3287026d49aa-kube-api-access-dlsjp\") pod \"payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.268700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.255416 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a469becb-c0d0-40cc-b426-3287026d49aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.273543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.270222 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f"] Oct 30 01:04:28.387947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.387664 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsjp\" (UniqueName: \"kubernetes.io/projected/a469becb-c0d0-40cc-b426-3287026d49aa-kube-api-access-dlsjp\") pod \"payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.440065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.423879 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:04:28.440065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.423937 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f"] Oct 30 01:04:28.440065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.424071 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:28.440065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.430661 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:04:28.446576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.446506 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:28.477543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.475282 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" event={"ID":"9370e3cd-a5c6-4b32-833b-68195297a32f","Type":"ContainerStarted","Data":"01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51"} Oct 30 01:04:28.481538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.478291 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" event={"ID":"338b099e-30c4-4f31-a6c8-8be499d56267","Type":"ContainerDied","Data":"7890e85ef7cbb3f35bf0fadb62ff5c4a04735866aa1bf7e33fa699f8fb770234"} Oct 30 01:04:28.481538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.478351 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7890e85ef7cbb3f35bf0fadb62ff5c4a04735866aa1bf7e33fa699f8fb770234" Oct 30 01:04:28.481538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.478422 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr" Oct 30 01:04:28.691098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.691065 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6w6\" (UniqueName: \"kubernetes.io/projected/fd181bda-aa31-4f73-9a62-c3485fa03d1f-kube-api-access-vb6w6\") pod \"payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:28.691727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.691707 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd181bda-aa31-4f73-9a62-c3485fa03d1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:28.795250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.793871 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6w6\" (UniqueName: \"kubernetes.io/projected/fd181bda-aa31-4f73-9a62-c3485fa03d1f-kube-api-access-vb6w6\") pod \"payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:28.795250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.793928 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd181bda-aa31-4f73-9a62-c3485fa03d1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:28.795250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.795093 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd181bda-aa31-4f73-9a62-c3485fa03d1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:28.802822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.802791 4905 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 01:04:28.802972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.802855 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 01:04:28.815881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.815855 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6w6\" (UniqueName: \"kubernetes.io/projected/fd181bda-aa31-4f73-9a62-c3485fa03d1f-kube-api-access-vb6w6\") pod \"payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:28.875416 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:28.874346 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda469becb_c0d0_40cc_b426_3287026d49aa.slice/crio-fa7c032cebba632e3bb029ac8d6763b5d55419f772d061fc3b283cdd2900ed37 WatchSource:0}: Error finding container fa7c032cebba632e3bb029ac8d6763b5d55419f772d061fc3b283cdd2900ed37: Status 404 returned error can't find the container with id fa7c032cebba632e3bb029ac8d6763b5d55419f772d061fc3b283cdd2900ed37 Oct 30 01:04:28.875416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:28.874842 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq"] Oct 30 01:04:29.066741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.065318 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:29.177478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.176864 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:04:29.177683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.177648 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:04:29.177762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.177695 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:04:29.179300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.179276 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:04:29.198910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.198887 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:04:29.208170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208146 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-socket\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208429 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-certs\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208492 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33cb223-2191-4064-87d5-0ce005175455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33cb223-2191-4064-87d5-0ce005175455-istiod-ca-cert\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208543 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a33cb223-2191-4064-87d5-0ce005175455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33cb223-2191-4064-87d5-0ce005175455-istio-podinfo\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208601 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-envoy\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208662 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-kube-api-access-z9ncm\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-kube-api-access-z9ncm\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208748 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-credential-socket\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208802 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-data\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.208848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.208831 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-istio-token\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-istio-token\") pod \"a33cb223-2191-4064-87d5-0ce005175455\" (UID: \"a33cb223-2191-4064-87d5-0ce005175455\") " Oct 30 01:04:29.209278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.209252 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-data" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.209465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.209443 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.209773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.209749 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-socket" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.209893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.209861 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-credential-socket" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.210211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.210188 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-certs" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.210775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.210754 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33cb223-2191-4064-87d5-0ce005175455-istiod-ca-cert" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:29.212070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.212049 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-istio-token" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:29.212174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.212107 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a33cb223-2191-4064-87d5-0ce005175455-istio-podinfo" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:29.212279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.212256 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-envoy" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.212348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.212283 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-kube-api-access-z9ncm" pod "a33cb223-2191-4064-87d5-0ce005175455" (UID: "a33cb223-2191-4064-87d5-0ce005175455"). InnerVolumeSpecName "kube-api-access-z9ncm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:29.310581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310512 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-certs\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.310761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310680 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-credential-socket\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.310761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310715 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4acc83b2-74c4-4499-a96b-3eecead8223d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4acc83b2-74c4-4499-a96b-3eecead8223d-istiod-ca-cert\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.310846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310760 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-kube-api-access-ds5xc\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-kube-api-access-ds5xc\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.310846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310761 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-certs" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.310846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310789 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-envoy\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.310965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310894 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-data\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.311006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310974 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-podinfo\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.311006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.310961 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-credential-socket" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.311080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.311003 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-token\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-token\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.311130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.311080 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-socket\") pod \"4acc83b2-74c4-4499-a96b-3eecead8223d\" (UID: \"4acc83b2-74c4-4499-a96b-3eecead8223d\") " Oct 30 01:04:29.311168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.311139 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acc83b2-74c4-4499-a96b-3eecead8223d-istiod-ca-cert" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:29.311481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.311440 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-socket" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.311481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.311451 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-data" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.312045 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312026 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312053 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312068 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312091 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312105 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33cb223-2191-4064-87d5-0ce005175455-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312119 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33cb223-2191-4064-87d5-0ce005175455-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312131 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312144 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312158 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ncm\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-kube-api-access-z9ncm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312176 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312188 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4acc83b2-74c4-4499-a96b-3eecead8223d-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312200 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33cb223-2191-4064-87d5-0ce005175455-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.312378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.312212 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33cb223-2191-4064-87d5-0ce005175455-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.313721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.313694 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-envoy" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:29.313815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.313729 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-kube-api-access-ds5xc" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "kube-api-access-ds5xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:29.313815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.313768 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-podinfo" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:29.314443 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.314427 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-token" pod "4acc83b2-74c4-4499-a96b-3eecead8223d" (UID: "4acc83b2-74c4-4499-a96b-3eecead8223d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:29.413905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.413873 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds5xc\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-kube-api-access-ds5xc\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.413905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.413898 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.413905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.413909 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.414140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.413918 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4acc83b2-74c4-4499-a96b-3eecead8223d-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:29.454593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.446342 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f"] Oct 30 01:04:29.456969 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:29.456941 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd181bda_aa31_4f73_9a62_c3485fa03d1f.slice/crio-c8f7a1baa99fcb0cfe8c1d57ceedd2e662eb92ca71f5459fb5cf86d976b1efd7 WatchSource:0}: Error finding container c8f7a1baa99fcb0cfe8c1d57ceedd2e662eb92ca71f5459fb5cf86d976b1efd7: Status 404 returned error can't find the container with id c8f7a1baa99fcb0cfe8c1d57ceedd2e662eb92ca71f5459fb5cf86d976b1efd7 Oct 30 01:04:29.490889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.490851 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" event={"ID":"a469becb-c0d0-40cc-b426-3287026d49aa","Type":"ContainerStarted","Data":"bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a"} Oct 30 01:04:29.491005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.490927 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" event={"ID":"a469becb-c0d0-40cc-b426-3287026d49aa","Type":"ContainerStarted","Data":"fa7c032cebba632e3bb029ac8d6763b5d55419f772d061fc3b283cdd2900ed37"} Oct 30 01:04:29.491005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.490993 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:29.492326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.492302 4905 generic.go:356] "Generic (PLEG): container finished" podID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerID="c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6" exitCode=0 Oct 30 01:04:29.492433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.492389 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" event={"ID":"4acc83b2-74c4-4499-a96b-3eecead8223d","Type":"ContainerDied","Data":"c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6"} Oct 30 01:04:29.492506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.492441 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" event={"ID":"4acc83b2-74c4-4499-a96b-3eecead8223d","Type":"ContainerDied","Data":"b8da34cd8eb5eb9f3d38dc1dd45ccc268cf405329a02e8021389e80748a2cfda"} Oct 30 01:04:29.492506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.492471 4905 scope.go:122] "RemoveContainer" containerID="c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6" Oct 30 01:04:29.492659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.492628 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw" Oct 30 01:04:29.494550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.494382 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" event={"ID":"fd181bda-aa31-4f73-9a62-c3485fa03d1f","Type":"ContainerStarted","Data":"c8f7a1baa99fcb0cfe8c1d57ceedd2e662eb92ca71f5459fb5cf86d976b1efd7"} Oct 30 01:04:29.496780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.496753 4905 generic.go:356] "Generic (PLEG): container finished" podID="a33cb223-2191-4064-87d5-0ce005175455" containerID="27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2" exitCode=0 Oct 30 01:04:29.497835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.497813 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" Oct 30 01:04:29.499567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.499547 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" event={"ID":"a33cb223-2191-4064-87d5-0ce005175455","Type":"ContainerDied","Data":"27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2"} Oct 30 01:04:29.499665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.499593 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p" event={"ID":"a33cb223-2191-4064-87d5-0ce005175455","Type":"ContainerDied","Data":"2c6a7a98803ec5d9befa88b389e338fb5feb02aa88699436ad6e12a706d80a84"} Oct 30 01:04:29.511404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.511386 4905 scope.go:122] "RemoveContainer" containerID="c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6" Oct 30 01:04:29.511740 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:29.511720 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6\": container with ID starting with c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6 not found: ID does not exist" containerID="c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6" Oct 30 01:04:29.511831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.511747 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6"} err="failed to get container status \"c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6\": rpc error: code = NotFound desc = could not find container \"c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6\": container with ID starting with c93490cb55f811ed26634105cbcf94d2b3dc93a8bdb61480596bba54d8f916f6 not found: ID does not exist" Oct 30 01:04:29.511831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.511762 4905 scope.go:122] "RemoveContainer" containerID="27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2" Oct 30 01:04:29.534042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.532652 4905 scope.go:122] "RemoveContainer" containerID="03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163" Oct 30 01:04:29.547266 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.541428 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" podStartSLOduration=1.541411232 podStartE2EDuration="1.541411232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:29.533588274 +0000 UTC m=+1554.476761468" watchObservedRunningTime="2026-10-30 01:04:29.541411232 +0000 UTC m=+1554.484584425" Oct 30 01:04:29.577258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.577234 4905 scope.go:122] "RemoveContainer" containerID="27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2" Oct 30 01:04:29.577723 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:29.577702 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2\": container with ID starting with 27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2 not found: ID does not exist" containerID="27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2" Oct 30 01:04:29.577870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.577844 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2"} err="failed to get container status \"27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2\": rpc error: code = NotFound desc = could not find container \"27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2\": container with ID starting with 27e287b3865f068d19fd655998bd3609ba787545c62c8a7099a44c6863ef1ed2 not found: ID does not exist" Oct 30 01:04:29.577966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.577956 4905 scope.go:122] "RemoveContainer" containerID="03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163" Oct 30 01:04:29.578323 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:29.578304 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163\": container with ID starting with 03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163 not found: ID does not exist" containerID="03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163" Oct 30 01:04:29.578505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.578483 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163"} err="failed to get container status \"03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163\": rpc error: code = NotFound desc = could not find container \"03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163\": container with ID starting with 03375c808a807188472703f0c6178b433e163032b98ba75d496cbfcfc24c1163 not found: ID does not exist" Oct 30 01:04:29.579452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.579438 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p"] Oct 30 01:04:29.583481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.583458 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-hpl6p"] Oct 30 01:04:29.615451 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.615403 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw"] Oct 30 01:04:29.621600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:29.621576 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-5ctlw"] Oct 30 01:04:30.161596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:30.157932 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:04:30.161596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:30.157988 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:04:30.510802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:30.510763 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" event={"ID":"fd181bda-aa31-4f73-9a62-c3485fa03d1f","Type":"ContainerStarted","Data":"15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce"} Oct 30 01:04:30.511292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:30.510858 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:31.158607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.158571 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:04:31.159018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.158985 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:04:31.196946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.196042 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" path="/var/lib/kubelet/pods/4acc83b2-74c4-4499-a96b-3eecead8223d/volumes" Oct 30 01:04:31.196946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.196606 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33cb223-2191-4064-87d5-0ce005175455" path="/var/lib/kubelet/pods/a33cb223-2191-4064-87d5-0ce005175455/volumes" Oct 30 01:04:31.344555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.344225 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" podStartSLOduration=3.344201647 podStartE2EDuration="3.344201647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:30.603073169 +0000 UTC m=+1555.546246362" watchObservedRunningTime="2026-10-30 01:04:31.344201647 +0000 UTC m=+1556.287374840" Oct 30 01:04:31.357548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.351578 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8901-a809d5"] Oct 30 01:04:31.357548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.352761 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" Oct 30 01:04:31.357548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.352780 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" Oct 30 01:04:31.357548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.352830 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" Oct 30 01:04:31.357548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.352838 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" Oct 30 01:04:31.357548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.353025 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acc83b2-74c4-4499-a96b-3eecead8223d" containerName="istio-proxy" Oct 30 01:04:31.357548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.353047 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" Oct 30 01:04:31.361541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.360564 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8901-a809d5"] Oct 30 01:04:31.361541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.360677 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:31.380736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.380711 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-98k27\"" Oct 30 01:04:31.445772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.445592 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnvg\" (UniqueName: \"kubernetes.io/projected/a85ce7f6-1f60-4c77-9911-463139c789c1-kube-api-access-cpnvg\") pod \"test-curl-8901-a809d5\" (UID: \"a85ce7f6-1f60-4c77-9911-463139c789c1\") " pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:31.555864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.555499 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnvg\" (UniqueName: \"kubernetes.io/projected/a85ce7f6-1f60-4c77-9911-463139c789c1-kube-api-access-cpnvg\") pod \"test-curl-8901-a809d5\" (UID: \"a85ce7f6-1f60-4c77-9911-463139c789c1\") " pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:31.585545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.578458 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnvg\" (UniqueName: \"kubernetes.io/projected/a85ce7f6-1f60-4c77-9911-463139c789c1-kube-api-access-cpnvg\") pod \"test-curl-8901-a809d5\" (UID: \"a85ce7f6-1f60-4c77-9911-463139c789c1\") " pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:31.682809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:31.682411 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:32.046548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:32.046436 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8901-a809d5"] Oct 30 01:04:32.082412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:32.078929 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:04:32.161549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:32.158158 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:04:32.161549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:32.158216 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:04:32.562478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:32.562436 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8901-a809d5" event={"ID":"a85ce7f6-1f60-4c77-9911-463139c789c1","Type":"ContainerStarted","Data":"5e06763accc1332cd49ee53f674d1fde9f804ec9b9c72abb18e212ee4f522ce8"} Oct 30 01:04:33.158171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:33.158125 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:04:33.158395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:33.158191 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:04:34.164345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:34.164282 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:04:34.165704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:34.165680 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:04:34.167224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:34.167198 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:04:34.752701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:34.752666 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk"] Oct 30 01:04:34.779682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:34.779648 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8"] Oct 30 01:04:35.616081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:35.616038 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8901-a809d5" event={"ID":"a85ce7f6-1f60-4c77-9911-463139c789c1","Type":"ContainerStarted","Data":"a176dafcc5b7489607ba22095deebf73db2a74def40aed86d2d9ca3bb73aeb78"} Oct 30 01:04:35.656512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:35.656454 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8901-a809d5" podStartSLOduration=2.110339313 podStartE2EDuration="4.656441414s" totalImagesPullingTime="2.546102101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:35.654036052 +0000 UTC m=+1560.597209244" watchObservedRunningTime="2026-10-30 01:04:35.656441414 +0000 UTC m=+1560.599614605" Oct 30 01:04:36.445544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.439201 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8901-a809d5"] Oct 30 01:04:36.446558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.446043 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8901-a809d5"] Oct 30 01:04:36.708097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.708022 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc"] Oct 30 01:04:36.708778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.708761 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a85ce7f6-1f60-4c77-9911-463139c789c1" containerName="test-curl-8901-a809d5" Oct 30 01:04:36.708853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.708780 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85ce7f6-1f60-4c77-9911-463139c789c1" containerName="test-curl-8901-a809d5" Oct 30 01:04:36.708853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.708822 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" Oct 30 01:04:36.708853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.708828 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" Oct 30 01:04:36.708949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.708932 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85ce7f6-1f60-4c77-9911-463139c789c1" containerName="test-curl-8901-a809d5" Oct 30 01:04:36.708949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.708944 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33cb223-2191-4064-87d5-0ce005175455" containerName="istio-proxy" Oct 30 01:04:36.725133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.724933 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.735578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.733354 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc"] Oct 30 01:04:36.755407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.755368 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:04:36.755618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.755415 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:04:36.755618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.755419 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:04:36.755618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.755477 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:04:36.814250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.814218 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.814432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.814292 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.814432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.814408 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vvm\" (UniqueName: \"kubernetes.io/projected/333062a5-2b10-4075-bcf0-c4e097d5f35f-kube-api-access-h8vvm\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.919692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.919646 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.920072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.920044 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vvm\" (UniqueName: \"kubernetes.io/projected/333062a5-2b10-4075-bcf0-c4e097d5f35f-kube-api-access-h8vvm\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.920553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.920504 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.950411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.922187 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.953331 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.925510 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:36.989659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:36.970825 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vvm\" (UniqueName: \"kubernetes.io/projected/333062a5-2b10-4075-bcf0-c4e097d5f35f-kube-api-access-h8vvm\") pod \"maas-api-revoke-keys-e2e-mt-869c367e-jq4xc\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:37.044974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:37.044934 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:37.357730 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:37.357698 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333062a5_2b10_4075_bcf0_c4e097d5f35f.slice/crio-867abe9259bb7968f74e50d69ca57a576af715ff585469cc5b9a95a4c1f88c46 WatchSource:0}: Error finding container 867abe9259bb7968f74e50d69ca57a576af715ff585469cc5b9a95a4c1f88c46: Status 404 returned error can't find the container with id 867abe9259bb7968f74e50d69ca57a576af715ff585469cc5b9a95a4c1f88c46 Oct 30 01:04:37.360480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:37.360454 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc"] Oct 30 01:04:37.636143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:37.636113 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" event={"ID":"333062a5-2b10-4075-bcf0-c4e097d5f35f","Type":"ContainerStarted","Data":"867abe9259bb7968f74e50d69ca57a576af715ff585469cc5b9a95a4c1f88c46"} Oct 30 01:04:37.636492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:37.636444 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8901-a809d5" podUID="a85ce7f6-1f60-4c77-9911-463139c789c1" containerName="test-curl-8901-a809d5" containerID="cri-o://a176dafcc5b7489607ba22095deebf73db2a74def40aed86d2d9ca3bb73aeb78" gracePeriod=2 Oct 30 01:04:38.697050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:38.696972 4905 generic.go:356] "Generic (PLEG): container finished" podID="333062a5-2b10-4075-bcf0-c4e097d5f35f" containerID="6b275f602faf7ad4d23e41be79112de5342844b88014257a19d95772f77e4f56" exitCode=0 Oct 30 01:04:38.697448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:38.697139 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" event={"ID":"333062a5-2b10-4075-bcf0-c4e097d5f35f","Type":"ContainerDied","Data":"6b275f602faf7ad4d23e41be79112de5342844b88014257a19d95772f77e4f56"} Oct 30 01:04:39.708235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:39.708203 4905 generic.go:356] "Generic (PLEG): container finished" podID="a85ce7f6-1f60-4c77-9911-463139c789c1" containerID="a176dafcc5b7489607ba22095deebf73db2a74def40aed86d2d9ca3bb73aeb78" exitCode=137 Oct 30 01:04:39.753540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:39.753351 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" podUID="1951e859-93e1-4b7e-bf19-94cd6f25cbee" containerName="payload-processing" containerID="cri-o://3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac" gracePeriod=25 Oct 30 01:04:39.780349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:39.780319 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" podUID="90ae684f-bc2b-4cbf-9f1d-194630e85870" containerName="payload-pre-processing" containerID="cri-o://70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887" gracePeriod=25 Oct 30 01:04:39.931820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:39.931788 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:39.989690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:39.989665 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:40.072261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.071285 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85ce7f6-1f60-4c77-9911-463139c789c1-kube-api-access-cpnvg\" (UniqueName: \"kubernetes.io/projected/a85ce7f6-1f60-4c77-9911-463139c789c1-kube-api-access-cpnvg\") pod \"a85ce7f6-1f60-4c77-9911-463139c789c1\" (UID: \"a85ce7f6-1f60-4c77-9911-463139c789c1\") " Oct 30 01:04:40.074854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.074828 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85ce7f6-1f60-4c77-9911-463139c789c1-kube-api-access-cpnvg" pod "a85ce7f6-1f60-4c77-9911-463139c789c1" (UID: "a85ce7f6-1f60-4c77-9911-463139c789c1"). InnerVolumeSpecName "kube-api-access-cpnvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:40.082342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.082312 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:04:40.172935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.172907 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-openshift-ca\") pod \"333062a5-2b10-4075-bcf0-c4e097d5f35f\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " Oct 30 01:04:40.173097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.173028 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-xks-ca\") pod \"333062a5-2b10-4075-bcf0-c4e097d5f35f\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " Oct 30 01:04:40.173258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.173245 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1951e859-93e1-4b7e-bf19-94cd6f25cbee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1951e859-93e1-4b7e-bf19-94cd6f25cbee-plugins-config-volume\") pod \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " Oct 30 01:04:40.173317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.173273 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1951e859-93e1-4b7e-bf19-94cd6f25cbee-kube-api-access-s2xd7\" (UniqueName: \"kubernetes.io/projected/1951e859-93e1-4b7e-bf19-94cd6f25cbee-kube-api-access-s2xd7\") pod \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\" (UID: \"1951e859-93e1-4b7e-bf19-94cd6f25cbee\") " Oct 30 01:04:40.173317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.173291 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333062a5-2b10-4075-bcf0-c4e097d5f35f-kube-api-access-h8vvm\" (UniqueName: \"kubernetes.io/projected/333062a5-2b10-4075-bcf0-c4e097d5f35f-kube-api-access-h8vvm\") pod \"333062a5-2b10-4075-bcf0-c4e097d5f35f\" (UID: \"333062a5-2b10-4075-bcf0-c4e097d5f35f\") " Oct 30 01:04:40.173746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.173731 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnvg\" (UniqueName: \"kubernetes.io/projected/a85ce7f6-1f60-4c77-9911-463139c789c1-kube-api-access-cpnvg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.174620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.174568 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1951e859-93e1-4b7e-bf19-94cd6f25cbee-plugins-config-volume" pod "1951e859-93e1-4b7e-bf19-94cd6f25cbee" (UID: "1951e859-93e1-4b7e-bf19-94cd6f25cbee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:40.175178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.175154 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-openshift-ca" pod "333062a5-2b10-4075-bcf0-c4e097d5f35f" (UID: "333062a5-2b10-4075-bcf0-c4e097d5f35f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:40.179197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.179172 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-xks-ca" pod "333062a5-2b10-4075-bcf0-c4e097d5f35f" (UID: "333062a5-2b10-4075-bcf0-c4e097d5f35f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:04:40.179293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.179194 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333062a5-2b10-4075-bcf0-c4e097d5f35f-kube-api-access-h8vvm" pod "333062a5-2b10-4075-bcf0-c4e097d5f35f" (UID: "333062a5-2b10-4075-bcf0-c4e097d5f35f"). InnerVolumeSpecName "kube-api-access-h8vvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:40.180491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.180471 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1951e859-93e1-4b7e-bf19-94cd6f25cbee-kube-api-access-s2xd7" pod "1951e859-93e1-4b7e-bf19-94cd6f25cbee" (UID: "1951e859-93e1-4b7e-bf19-94cd6f25cbee"). InnerVolumeSpecName "kube-api-access-s2xd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:40.182692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.182674 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:04:40.274578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.274497 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ae684f-bc2b-4cbf-9f1d-194630e85870-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ae684f-bc2b-4cbf-9f1d-194630e85870-plugins-config-volume\") pod \"90ae684f-bc2b-4cbf-9f1d-194630e85870\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " Oct 30 01:04:40.274919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.274899 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ae684f-bc2b-4cbf-9f1d-194630e85870-kube-api-access-fktlx\" (UniqueName: \"kubernetes.io/projected/90ae684f-bc2b-4cbf-9f1d-194630e85870-kube-api-access-fktlx\") pod \"90ae684f-bc2b-4cbf-9f1d-194630e85870\" (UID: \"90ae684f-bc2b-4cbf-9f1d-194630e85870\") " Oct 30 01:04:40.275680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.275659 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.275762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.275688 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1951e859-93e1-4b7e-bf19-94cd6f25cbee-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.275762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.275705 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xd7\" (UniqueName: \"kubernetes.io/projected/1951e859-93e1-4b7e-bf19-94cd6f25cbee-kube-api-access-s2xd7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.275762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.275718 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8vvm\" (UniqueName: \"kubernetes.io/projected/333062a5-2b10-4075-bcf0-c4e097d5f35f-kube-api-access-h8vvm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.275762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.275731 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333062a5-2b10-4075-bcf0-c4e097d5f35f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.276992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.276963 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ae684f-bc2b-4cbf-9f1d-194630e85870-plugins-config-volume" pod "90ae684f-bc2b-4cbf-9f1d-194630e85870" (UID: "90ae684f-bc2b-4cbf-9f1d-194630e85870"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:40.278837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.278812 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ae684f-bc2b-4cbf-9f1d-194630e85870-kube-api-access-fktlx" pod "90ae684f-bc2b-4cbf-9f1d-194630e85870" (UID: "90ae684f-bc2b-4cbf-9f1d-194630e85870"). InnerVolumeSpecName "kube-api-access-fktlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:40.377509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.377473 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktlx\" (UniqueName: \"kubernetes.io/projected/90ae684f-bc2b-4cbf-9f1d-194630e85870-kube-api-access-fktlx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.377509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.377502 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ae684f-bc2b-4cbf-9f1d-194630e85870-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.537014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.533486 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:04:40.718845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.718814 4905 generic.go:356] "Generic (PLEG): container finished" podID="1951e859-93e1-4b7e-bf19-94cd6f25cbee" containerID="3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac" exitCode=0 Oct 30 01:04:40.719317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.718859 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" Oct 30 01:04:40.719317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.718904 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" event={"ID":"1951e859-93e1-4b7e-bf19-94cd6f25cbee","Type":"ContainerDied","Data":"3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac"} Oct 30 01:04:40.719317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.718957 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk" event={"ID":"1951e859-93e1-4b7e-bf19-94cd6f25cbee","Type":"ContainerDied","Data":"58ba1dae65d7dd73ce1e57860c355e37d1b01acf2707f5818fef8f0a420d14a5"} Oct 30 01:04:40.719317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.718987 4905 scope.go:122] "RemoveContainer" containerID="3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac" Oct 30 01:04:40.720938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.720916 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:40.722651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.722636 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" Oct 30 01:04:40.722778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.722686 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc" event={"ID":"333062a5-2b10-4075-bcf0-c4e097d5f35f","Type":"ContainerDied","Data":"867abe9259bb7968f74e50d69ca57a576af715ff585469cc5b9a95a4c1f88c46"} Oct 30 01:04:40.722778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.722733 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="867abe9259bb7968f74e50d69ca57a576af715ff585469cc5b9a95a4c1f88c46" Oct 30 01:04:40.724686 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.724660 4905 generic.go:356] "Generic (PLEG): container finished" podID="90ae684f-bc2b-4cbf-9f1d-194630e85870" containerID="70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887" exitCode=0 Oct 30 01:04:40.724803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.724708 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" Oct 30 01:04:40.724803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.724767 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" event={"ID":"90ae684f-bc2b-4cbf-9f1d-194630e85870","Type":"ContainerDied","Data":"70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887"} Oct 30 01:04:40.724916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.724805 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8" event={"ID":"90ae684f-bc2b-4cbf-9f1d-194630e85870","Type":"ContainerDied","Data":"fed1f00f24d5a578dba925cc177a041c62f41e8d4695aa2d723eac01c8dc2f54"} Oct 30 01:04:40.736364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.736336 4905 scope.go:122] "RemoveContainer" containerID="3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac" Oct 30 01:04:40.736844 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:40.736694 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac\": container with ID starting with 3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac not found: ID does not exist" containerID="3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac" Oct 30 01:04:40.736844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.736723 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac"} err="failed to get container status \"3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac\": rpc error: code = NotFound desc = could not find container \"3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac\": container with ID starting with 3c626e86fe057632a4c11a3e80c46d8ad94ede4910280f9dacacda8e4a27d3ac not found: ID does not exist" Oct 30 01:04:40.736844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.736745 4905 scope.go:122] "RemoveContainer" containerID="a176dafcc5b7489607ba22095deebf73db2a74def40aed86d2d9ca3bb73aeb78" Oct 30 01:04:40.750131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.750110 4905 scope.go:122] "RemoveContainer" containerID="70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887" Oct 30 01:04:40.751688 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:40.751507 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8901-a809d5\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="a85ce7f6-1f60-4c77-9911-463139c789c1" pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:40.761114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.761095 4905 scope.go:122] "RemoveContainer" containerID="70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887" Oct 30 01:04:40.761444 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:40.761421 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887\": container with ID starting with 70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887 not found: ID does not exist" containerID="70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887" Oct 30 01:04:40.761502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.761455 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887"} err="failed to get container status \"70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887\": rpc error: code = NotFound desc = could not find container \"70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887\": container with ID starting with 70c5e9af4e849e1e02e7b0bee8a7256acd47114a1c939d22ac9bb5fa0f442887 not found: ID does not exist" Oct 30 01:04:40.794199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.794122 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk"] Oct 30 01:04:40.799040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.799018 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee0869-84499bcd66-lhvgk"] Oct 30 01:04:40.849863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.849834 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8"] Oct 30 01:04:40.856172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:40.856150 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0869-65c7bfbbf9-kspq8"] Oct 30 01:04:40.881911 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:40.881882 4905 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8901-a809d5\" is forbidden: User \"system:node:ip-10-0-130-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-171.ec2.internal' and this object" podUID="a85ce7f6-1f60-4c77-9911-463139c789c1" pod="opendatahub/test-curl-8901-a809d5" Oct 30 01:04:41.192623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:41.192507 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1951e859-93e1-4b7e-bf19-94cd6f25cbee" path="/var/lib/kubelet/pods/1951e859-93e1-4b7e-bf19-94cd6f25cbee/volumes" Oct 30 01:04:41.193018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:41.193003 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ae684f-bc2b-4cbf-9f1d-194630e85870" path="/var/lib/kubelet/pods/90ae684f-bc2b-4cbf-9f1d-194630e85870/volumes" Oct 30 01:04:41.193350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:41.193337 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85ce7f6-1f60-4c77-9911-463139c789c1" path="/var/lib/kubelet/pods/a85ce7f6-1f60-4c77-9911-463139c789c1/volumes" Oct 30 01:04:41.523238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:41.523211 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:04:42.068993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.068956 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8"] Oct 30 01:04:42.070098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070075 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1951e859-93e1-4b7e-bf19-94cd6f25cbee" containerName="payload-processing" Oct 30 01:04:42.070098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070098 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1951e859-93e1-4b7e-bf19-94cd6f25cbee" containerName="payload-processing" Oct 30 01:04:42.070292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070119 4905 cpu_manager.go:414] "containerMap: removing container" podUID="90ae684f-bc2b-4cbf-9f1d-194630e85870" containerName="payload-pre-processing" Oct 30 01:04:42.070292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070127 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ae684f-bc2b-4cbf-9f1d-194630e85870" containerName="payload-pre-processing" Oct 30 01:04:42.070292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070190 4905 cpu_manager.go:414] "containerMap: removing container" podUID="333062a5-2b10-4075-bcf0-c4e097d5f35f" containerName="revoke-keys" Oct 30 01:04:42.070292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070197 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333062a5-2b10-4075-bcf0-c4e097d5f35f" containerName="revoke-keys" Oct 30 01:04:42.070504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070399 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="333062a5-2b10-4075-bcf0-c4e097d5f35f" containerName="revoke-keys" Oct 30 01:04:42.070504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070418 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1951e859-93e1-4b7e-bf19-94cd6f25cbee" containerName="payload-processing" Oct 30 01:04:42.070504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.070426 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ae684f-bc2b-4cbf-9f1d-194630e85870" containerName="payload-pre-processing" Oct 30 01:04:42.076532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.076502 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.092024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.091783 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8"] Oct 30 01:04:42.132029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.127961 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:04:42.132029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.128181 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:04:42.132029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.128281 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:04:42.173191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.173089 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6w4xf\"" Oct 30 01:04:42.173191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.173038 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2ca20f-serving-cert\"" Oct 30 01:04:42.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.231137 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-metrics-certs\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.231338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.231287 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9tz2\" (UniqueName: \"kubernetes.io/projected/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-kube-api-access-p9tz2\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.231405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.231382 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.231474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.231457 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.347598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.338624 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-metrics-certs\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.347598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.338880 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9tz2\" (UniqueName: \"kubernetes.io/projected/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-kube-api-access-p9tz2\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.347598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.339024 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.347598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.339127 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.347598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.339971 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.371064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.369323 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.382458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.382116 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-metrics-certs\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.389732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.383191 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9tz2\" (UniqueName: \"kubernetes.io/projected/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-kube-api-access-p9tz2\") pod \"maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.423282 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:42.423246 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:04:42.434737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.434712 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:42.630336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.630237 4905 scope.go:122] "RemoveContainer" containerID="0f5ffa99f5590a0e73f0a5a8e9ff5f0f787029f27cc6c2fb3462c41e9c53295d" Oct 30 01:04:42.647176 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.647095 4905 scope.go:122] "RemoveContainer" containerID="ee6f56dbeb77c66c42debd34b014cbf70b854fa97aec95e06464d93828d6deab" Oct 30 01:04:42.664808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.664781 4905 scope.go:122] "RemoveContainer" containerID="14daf8e9b770e2a85a8c2f26540e72dda000db9f56bc95beb435ffd9519a90f3" Oct 30 01:04:42.677179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.677156 4905 scope.go:122] "RemoveContainer" containerID="186a8eb8fb7aa959ef10ad71a638585b7a5d102aa104aba89395e043cc864272" Oct 30 01:04:42.687152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.687135 4905 scope.go:122] "RemoveContainer" containerID="761f06935d00ce5a0d8b37fa062384e3a7bcea91f69b0790bffa4a0a498108d3" Oct 30 01:04:42.696407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.696390 4905 scope.go:122] "RemoveContainer" containerID="1aba5411c9c58a0f8d8a8e25d2ed661938822ff93a07d3945da37ffc73878990" Oct 30 01:04:42.706009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.705988 4905 scope.go:122] "RemoveContainer" containerID="e7d519cb376e911df829159102d7aaa30d50837bd587796981de57f9bc9ab143" Oct 30 01:04:42.750372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:42.750344 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8"] Oct 30 01:04:43.800874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:43.800838 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" event={"ID":"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e","Type":"ContainerStarted","Data":"2f498efcf9d3a2e50d30c744bfd22eae4689718daf93bad7c4e0ea48d0410ba7"} Oct 30 01:04:44.812605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:44.812565 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" event={"ID":"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e","Type":"ContainerStarted","Data":"d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2"} Oct 30 01:04:44.813107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:44.812775 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:44.858570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:44.857338 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" podStartSLOduration=1.127497101 podStartE2EDuration="2.857319081s" totalImagesPullingTime="1.72982198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:44.853358629 +0000 UTC m=+1569.796531822" watchObservedRunningTime="2026-10-30 01:04:44.857319081 +0000 UTC m=+1569.800492274" Oct 30 01:04:47.148092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:47.148057 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9"] Oct 30 01:04:47.194648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:47.194622 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67"] Oct 30 01:04:50.836549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:50.835673 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:04:52.048545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.046102 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2"] Oct 30 01:04:52.059152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.059122 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e98016-6c7l2"] Oct 30 01:04:52.150543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.149607 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" podUID="15671937-b5d5-4ae5-9aef-53f1a3892717" containerName="payload-processing" containerID="cri-o://5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573" gracePeriod=25 Oct 30 01:04:52.195500 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.195465 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" podUID="250d7d06-3e88-4efb-8190-07e6a776ddac" containerName="payload-pre-processing" containerID="cri-o://07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67" gracePeriod=25 Oct 30 01:04:52.615735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.613754 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:04:52.773762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.773727 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:04:52.809587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.803295 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15671937-b5d5-4ae5-9aef-53f1a3892717-kube-api-access-hws4l\" (UniqueName: \"kubernetes.io/projected/15671937-b5d5-4ae5-9aef-53f1a3892717-kube-api-access-hws4l\") pod \"15671937-b5d5-4ae5-9aef-53f1a3892717\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " Oct 30 01:04:52.809587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.803504 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15671937-b5d5-4ae5-9aef-53f1a3892717-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15671937-b5d5-4ae5-9aef-53f1a3892717-plugins-config-volume\") pod \"15671937-b5d5-4ae5-9aef-53f1a3892717\" (UID: \"15671937-b5d5-4ae5-9aef-53f1a3892717\") " Oct 30 01:04:52.809587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.804309 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15671937-b5d5-4ae5-9aef-53f1a3892717-plugins-config-volume" pod "15671937-b5d5-4ae5-9aef-53f1a3892717" (UID: "15671937-b5d5-4ae5-9aef-53f1a3892717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:52.809587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.807262 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15671937-b5d5-4ae5-9aef-53f1a3892717-kube-api-access-hws4l" pod "15671937-b5d5-4ae5-9aef-53f1a3892717" (UID: "15671937-b5d5-4ae5-9aef-53f1a3892717"). InnerVolumeSpecName "kube-api-access-hws4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:52.905065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.904969 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250d7d06-3e88-4efb-8190-07e6a776ddac-kube-api-access-skklr\" (UniqueName: \"kubernetes.io/projected/250d7d06-3e88-4efb-8190-07e6a776ddac-kube-api-access-skklr\") pod \"250d7d06-3e88-4efb-8190-07e6a776ddac\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " Oct 30 01:04:52.905229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.905123 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250d7d06-3e88-4efb-8190-07e6a776ddac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250d7d06-3e88-4efb-8190-07e6a776ddac-plugins-config-volume\") pod \"250d7d06-3e88-4efb-8190-07e6a776ddac\" (UID: \"250d7d06-3e88-4efb-8190-07e6a776ddac\") " Oct 30 01:04:52.905566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.905532 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250d7d06-3e88-4efb-8190-07e6a776ddac-plugins-config-volume" pod "250d7d06-3e88-4efb-8190-07e6a776ddac" (UID: "250d7d06-3e88-4efb-8190-07e6a776ddac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:52.905820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.905803 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15671937-b5d5-4ae5-9aef-53f1a3892717-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:52.905879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.905821 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250d7d06-3e88-4efb-8190-07e6a776ddac-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:52.905879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.905830 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hws4l\" (UniqueName: \"kubernetes.io/projected/15671937-b5d5-4ae5-9aef-53f1a3892717-kube-api-access-hws4l\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:52.907300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.907277 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250d7d06-3e88-4efb-8190-07e6a776ddac-kube-api-access-skklr" pod "250d7d06-3e88-4efb-8190-07e6a776ddac" (UID: "250d7d06-3e88-4efb-8190-07e6a776ddac"). InnerVolumeSpecName "kube-api-access-skklr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:52.936544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.926212 4905 generic.go:356] "Generic (PLEG): container finished" podID="250d7d06-3e88-4efb-8190-07e6a776ddac" containerID="07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67" exitCode=0 Oct 30 01:04:52.936544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.926454 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" event={"ID":"250d7d06-3e88-4efb-8190-07e6a776ddac","Type":"ContainerDied","Data":"07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67"} Oct 30 01:04:52.936544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.926509 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" event={"ID":"250d7d06-3e88-4efb-8190-07e6a776ddac","Type":"ContainerDied","Data":"a88500a7bf69c75900223af9a523e465e0f297afdf4c52d8cc57cbb0b6dcedae"} Oct 30 01:04:52.936544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.926561 4905 scope.go:122] "RemoveContainer" containerID="07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67" Oct 30 01:04:52.936544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.926710 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67" Oct 30 01:04:52.949794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.942293 4905 generic.go:356] "Generic (PLEG): container finished" podID="15671937-b5d5-4ae5-9aef-53f1a3892717" containerID="5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573" exitCode=0 Oct 30 01:04:52.949794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.942463 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" event={"ID":"15671937-b5d5-4ae5-9aef-53f1a3892717","Type":"ContainerDied","Data":"5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573"} Oct 30 01:04:52.949794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.942506 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" event={"ID":"15671937-b5d5-4ae5-9aef-53f1a3892717","Type":"ContainerDied","Data":"f49b0722e13b719d238f437a8e296f6bcfa9ce3669981d23c943fc9947c34127"} Oct 30 01:04:52.949794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.942703 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9" Oct 30 01:04:52.962180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.962164 4905 scope.go:122] "RemoveContainer" containerID="07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67" Oct 30 01:04:52.962462 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:52.962446 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67\": container with ID starting with 07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67 not found: ID does not exist" containerID="07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67" Oct 30 01:04:52.962542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.962470 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67"} err="failed to get container status \"07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67\": rpc error: code = NotFound desc = could not find container \"07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67\": container with ID starting with 07c8473dd25c00d4d041c9f65d76c2a2c99e8ce8df728139ad5d051c959faa67 not found: ID does not exist" Oct 30 01:04:52.962542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.962488 4905 scope.go:122] "RemoveContainer" containerID="5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573" Oct 30 01:04:52.972095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.972079 4905 scope.go:122] "RemoveContainer" containerID="5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573" Oct 30 01:04:52.972372 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:04:52.972354 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573\": container with ID starting with 5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573 not found: ID does not exist" containerID="5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573" Oct 30 01:04:52.972429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.972378 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573"} err="failed to get container status \"5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573\": rpc error: code = NotFound desc = could not find container \"5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573\": container with ID starting with 5e4e0ebb47baa3a4e3ada7b8f2994c99bb694dd383f0501f25fa1738eec53573 not found: ID does not exist" Oct 30 01:04:52.994738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:52.994714 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67"] Oct 30 01:04:53.002179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:53.002142 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-869c367e-7d968c6755-p4b67"] Oct 30 01:04:53.007991 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:53.007965 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skklr\" (UniqueName: \"kubernetes.io/projected/250d7d06-3e88-4efb-8190-07e6a776ddac-kube-api-access-skklr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:04:53.054744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:53.054698 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9"] Oct 30 01:04:53.059156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:53.059128 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-869c367e-7d54b86f54-jhtp9"] Oct 30 01:04:53.191717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:53.191688 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15671937-b5d5-4ae5-9aef-53f1a3892717" path="/var/lib/kubelet/pods/15671937-b5d5-4ae5-9aef-53f1a3892717/volumes" Oct 30 01:04:53.192150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:53.192136 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a79bc14-c169-4816-8ec8-ea7df7bc2125" path="/var/lib/kubelet/pods/1a79bc14-c169-4816-8ec8-ea7df7bc2125/volumes" Oct 30 01:04:53.192548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:53.192530 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250d7d06-3e88-4efb-8190-07e6a776ddac" path="/var/lib/kubelet/pods/250d7d06-3e88-4efb-8190-07e6a776ddac/volumes" Oct 30 01:04:54.712625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.712590 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97"] Oct 30 01:04:54.713801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.713762 4905 cpu_manager.go:414] "containerMap: removing container" podUID="15671937-b5d5-4ae5-9aef-53f1a3892717" containerName="payload-processing" Oct 30 01:04:54.713801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.713798 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15671937-b5d5-4ae5-9aef-53f1a3892717" containerName="payload-processing" Oct 30 01:04:54.713979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.713866 4905 cpu_manager.go:414] "containerMap: removing container" podUID="250d7d06-3e88-4efb-8190-07e6a776ddac" containerName="payload-pre-processing" Oct 30 01:04:54.713979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.713875 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250d7d06-3e88-4efb-8190-07e6a776ddac" containerName="payload-pre-processing" Oct 30 01:04:54.714094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.714043 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="250d7d06-3e88-4efb-8190-07e6a776ddac" containerName="payload-pre-processing" Oct 30 01:04:54.714094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.714063 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="15671937-b5d5-4ae5-9aef-53f1a3892717" containerName="payload-processing" Oct 30 01:04:54.718392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.718374 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.739491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.739462 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97"] Oct 30 01:04:54.745604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.745572 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-bdf6f3-serving-cert\"" Oct 30 01:04:54.771264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.771227 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.771421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.771310 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2cx\" (UniqueName: \"kubernetes.io/projected/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-kube-api-access-fw2cx\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.771498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.771419 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.771498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.771478 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.801302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.801268 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc"] Oct 30 01:04:54.807029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.807000 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-869c367e-jq4xc"] Oct 30 01:04:54.873119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.873081 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.873312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.873172 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.873374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.873330 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.873439 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.873405 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2cx\" (UniqueName: \"kubernetes.io/projected/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-kube-api-access-fw2cx\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.873958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.873934 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.875919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.875890 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.876281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.876260 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:54.894057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:54.894020 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2cx\" (UniqueName: \"kubernetes.io/projected/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-kube-api-access-fw2cx\") pod \"maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:55.041723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:55.041690 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:55.195195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:55.195158 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333062a5-2b10-4075-bcf0-c4e097d5f35f" path="/var/lib/kubelet/pods/333062a5-2b10-4075-bcf0-c4e097d5f35f/volumes" Oct 30 01:04:55.364503 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:04:55.364467 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede6ca67_abbc_4bda_8b14_2f3ac1eac173.slice/crio-3b124f2ffab6abc2f81e5644f6ed97d846af46d28ecff2fbb32e00e53aa364b3 WatchSource:0}: Error finding container 3b124f2ffab6abc2f81e5644f6ed97d846af46d28ecff2fbb32e00e53aa364b3: Status 404 returned error can't find the container with id 3b124f2ffab6abc2f81e5644f6ed97d846af46d28ecff2fbb32e00e53aa364b3 Oct 30 01:04:55.366885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:55.366864 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97"] Oct 30 01:04:56.042819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:56.042768 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" event={"ID":"ede6ca67-abbc-4bda-8b14-2f3ac1eac173","Type":"ContainerStarted","Data":"a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f"} Oct 30 01:04:56.043249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:56.042872 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" event={"ID":"ede6ca67-abbc-4bda-8b14-2f3ac1eac173","Type":"ContainerStarted","Data":"3b124f2ffab6abc2f81e5644f6ed97d846af46d28ecff2fbb32e00e53aa364b3"} Oct 30 01:04:56.043249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:56.043028 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:04:56.085404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:56.085350 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" podStartSLOduration=1.587582372 podStartE2EDuration="2.085336244s" totalImagesPullingTime="497.753872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:56.084222269 +0000 UTC m=+1581.027395462" watchObservedRunningTime="2026-10-30 01:04:56.085336244 +0000 UTC m=+1581.028509435" Oct 30 01:04:57.117671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:57.117632 4905 patch_prober.go:29] interesting pod/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:04:57.118218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:57.117697 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:59.079293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:59.079256 4905 generic.go:356] "Generic (PLEG): container finished" podID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerID="db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42" exitCode=0 Oct 30 01:04:59.079895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:59.079320 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" event={"ID":"fffa3510-3a48-4aff-b299-584ce5e7a0ea","Type":"ContainerDied","Data":"db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42"} Oct 30 01:04:59.079970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:59.079950 4905 scope.go:122] "RemoveContainer" containerID="db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42" Oct 30 01:04:59.081690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:59.081660 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:59.081790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:59.081730 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:04:59.081790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:04:59.081760 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:00.088907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:00.088872 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" event={"ID":"fffa3510-3a48-4aff-b299-584ce5e7a0ea","Type":"ContainerStarted","Data":"44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62"} Oct 30 01:05:00.586181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:00.585993 4905 patch_prober.go:29] interesting pod/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 01:05:00.586181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:00.586051 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 01:05:00.586472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:00.585685 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:05:01.584681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:01.584636 4905 patch_prober.go:29] interesting pod/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 01:05:01.585208 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:01.584713 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 01:05:02.063170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:02.063136 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:05:02.584696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:02.584658 4905 patch_prober.go:29] interesting pod/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 01:05:02.585105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:02.584707 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 01:05:03.585269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:03.585233 4905 patch_prober.go:29] interesting pod/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 01:05:03.585793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:03.585295 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 01:05:04.584787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:04.584746 4905 patch_prober.go:29] interesting pod/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 01:05:04.584975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:04.584808 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 01:05:04.873513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:04.873426 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m"] Oct 30 01:05:04.881781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:04.881698 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:04.904804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:04.900088 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bd9cc5-openshift-default-dockercfg-qjhxg\"" Oct 30 01:05:04.920255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:04.920222 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m"] Oct 30 01:05:05.021830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.021798 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-certs\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.021856 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-token\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.021955 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-socket\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.022060 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-credential-socket\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.022113 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-podinfo\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.022205 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-envoy\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.022265 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-data\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.022340 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjpt\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-kube-api-access-9cjpt\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.022420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.022406 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istiod-ca-cert\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.135623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.135466 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-token\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.268681 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-socket\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.268810 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-credential-socket\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.268870 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-podinfo\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.269007 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-envoy\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.269075 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-data\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.269252 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjpt\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-kube-api-access-9cjpt\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.269334 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istiod-ca-cert\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.290554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.269371 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-certs\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.320564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.291488 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-certs\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.320564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.291863 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-socket\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.320564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.292119 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-credential-socket\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.320564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.313190 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-token\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.342565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.338650 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-data\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.358561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.350898 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-envoy\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.397085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.392190 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istiod-ca-cert\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.399549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.399489 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-podinfo\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.401297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.401271 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjpt\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-kube-api-access-9cjpt\") pod \"e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.498449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.498417 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:05.596761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.584988 4905 patch_prober.go:29] interesting pod/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 01:05:05.596761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.585044 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 01:05:05.847126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.847078 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:05.847237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.847171 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:05.847237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.847213 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:05.850211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:05.849696 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m"] Oct 30 01:05:06.491720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.491681 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" event={"ID":"02bb569b-2dc4-48ec-9d22-6b04e2d0f088","Type":"ContainerStarted","Data":"a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e"} Oct 30 01:05:06.492320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.492293 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" event={"ID":"02bb569b-2dc4-48ec-9d22-6b04e2d0f088","Type":"ContainerStarted","Data":"0bc5cdd8ec71cd7d55a404b9f10ef7f6fa25e725d9a3ea4113e0f07f339f0cad"} Oct 30 01:05:06.512694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.503804 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:06.512694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.503922 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:05:06.512694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.503963 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:05:06.589868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.589841 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:05:06.591814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.590406 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:05:06.592200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.592185 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:05:06.644604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.639185 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podStartSLOduration=2.639165867 podStartE2EDuration="2.639165867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:06.539855024 +0000 UTC m=+1591.483028218" watchObservedRunningTime="2026-10-30 01:05:06.639165867 +0000 UTC m=+1591.582339062" Oct 30 01:05:06.654307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.654198 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x"] Oct 30 01:05:06.667047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:06.662743 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30c405-c299x"] Oct 30 01:05:07.191947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:07.191918 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19020704-51e4-4d08-a073-efcca1c0a35f" path="/var/lib/kubelet/pods/19020704-51e4-4d08-a073-efcca1c0a35f/volumes" Oct 30 01:05:07.504046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:07.504016 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:08.561843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:08.561814 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:08.563877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:08.563846 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:05:11.460586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.460554 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q"] Oct 30 01:05:11.466820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.466793 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:11.487042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.487013 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q"] Oct 30 01:05:11.494405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.493792 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bd9cc5\"" Oct 30 01:05:11.494405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.493897 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bd9cc5-dockercfg-rkqwl\"" Oct 30 01:05:11.604541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.603401 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5691598-7eb7-449d-a4cf-b905f1db9cd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:11.604541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.603489 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9fj\" (UniqueName: \"kubernetes.io/projected/c5691598-7eb7-449d-a4cf-b905f1db9cd5-kube-api-access-4k9fj\") pod \"payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:11.675543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.674597 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx"] Oct 30 01:05:11.709611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.708392 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9fj\" (UniqueName: \"kubernetes.io/projected/c5691598-7eb7-449d-a4cf-b905f1db9cd5-kube-api-access-4k9fj\") pod \"payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:11.709611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.708804 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5691598-7eb7-449d-a4cf-b905f1db9cd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:11.713709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.710128 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx"] Oct 30 01:05:11.713709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.710235 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:11.762545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.757829 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5691598-7eb7-449d-a4cf-b905f1db9cd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:11.806482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.806317 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9fj\" (UniqueName: \"kubernetes.io/projected/c5691598-7eb7-449d-a4cf-b905f1db9cd5-kube-api-access-4k9fj\") pod \"payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:11.847542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.830986 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:11.847542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.831128 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8tbk\" (UniqueName: \"kubernetes.io/projected/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-kube-api-access-s8tbk\") pod \"payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:11.933432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.933382 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:11.933700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.933506 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8tbk\" (UniqueName: \"kubernetes.io/projected/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-kube-api-access-s8tbk\") pod \"payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:11.934957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.934929 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:11.970615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:11.968383 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8tbk\" (UniqueName: \"kubernetes.io/projected/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-kube-api-access-s8tbk\") pod \"payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:12.083216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.083183 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:12.220913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.220819 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:12.424387 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:12.424343 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5691598_7eb7_449d_a4cf_b905f1db9cd5.slice/crio-3860720456d5b21960882d859206a7d3e528e9612ca6812879609a3a26aa988c WatchSource:0}: Error finding container 3860720456d5b21960882d859206a7d3e528e9612ca6812879609a3a26aa988c: Status 404 returned error can't find the container with id 3860720456d5b21960882d859206a7d3e528e9612ca6812879609a3a26aa988c Oct 30 01:05:12.466765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.465681 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q"] Oct 30 01:05:12.475488 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:12.475419 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:05:12.669307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.664884 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx"] Oct 30 01:05:12.671730 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:12.671695 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a9ee6a_f443_44b4_b8a9_32ed357beae9.slice/crio-44d294e45fbeab9a6054f8994e5cb2d2c2c123797c83368a2e010b8a1a076ac8 WatchSource:0}: Error finding container 44d294e45fbeab9a6054f8994e5cb2d2c2c123797c83368a2e010b8a1a076ac8: Status 404 returned error can't find the container with id 44d294e45fbeab9a6054f8994e5cb2d2c2c123797c83368a2e010b8a1a076ac8 Oct 30 01:05:12.869211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.869112 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" event={"ID":"f5a9ee6a-f443-44b4-b8a9-32ed357beae9","Type":"ContainerStarted","Data":"fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963"} Oct 30 01:05:12.869211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.869176 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" event={"ID":"f5a9ee6a-f443-44b4-b8a9-32ed357beae9","Type":"ContainerStarted","Data":"44d294e45fbeab9a6054f8994e5cb2d2c2c123797c83368a2e010b8a1a076ac8"} Oct 30 01:05:12.869969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.869899 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:12.871990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.871868 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" event={"ID":"c5691598-7eb7-449d-a4cf-b905f1db9cd5","Type":"ContainerStarted","Data":"5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8"} Oct 30 01:05:12.871990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.871916 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" event={"ID":"c5691598-7eb7-449d-a4cf-b905f1db9cd5","Type":"ContainerStarted","Data":"3860720456d5b21960882d859206a7d3e528e9612ca6812879609a3a26aa988c"} Oct 30 01:05:12.871990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.871944 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:12.915027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.914922 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" podStartSLOduration=1.914906556 podStartE2EDuration="1.914906556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:12.910774181 +0000 UTC m=+1597.853947374" watchObservedRunningTime="2026-10-30 01:05:12.914906556 +0000 UTC m=+1597.858079747" Oct 30 01:05:12.961983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:12.961928 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" podStartSLOduration=1.9619124669999999 podStartE2EDuration="1.961912467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:12.957568367 +0000 UTC m=+1597.900741559" watchObservedRunningTime="2026-10-30 01:05:12.961912467 +0000 UTC m=+1597.905085658" Oct 30 01:05:14.170545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:14.159893 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944"] Oct 30 01:05:14.170545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:14.160140 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" containerID="cri-o://f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a" gracePeriod=30 Oct 30 01:05:15.373822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.373784 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx"] Oct 30 01:05:15.383675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.383642 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.418307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.417690 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49d9f1-openshift-default-dockercfg-svv8m\"" Oct 30 01:05:15.418307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.418242 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx"] Oct 30 01:05:15.588596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.584567 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0996ac-76a8-4320-ad1d-7cad3c396659-istiod-ca-cert\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.679663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.679568 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-certs\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.685789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.685752 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-podinfo\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.686183 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.686150 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-credential-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.692734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.692703 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.693131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.693106 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw5nq\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-kube-api-access-pw5nq\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.693450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.693422 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-envoy\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.693728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.693708 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-token\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.693979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.693964 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-data\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.798581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.798535 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw5nq\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-kube-api-access-pw5nq\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.807146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.807100 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-envoy\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.810541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.807953 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-token\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.831999 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-data\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.832051 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0996ac-76a8-4320-ad1d-7cad3c396659-istiod-ca-cert\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.832126 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-certs\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.832197 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-podinfo\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.832265 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-credential-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.832325 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.832854 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.833093 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-data\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.833734 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0996ac-76a8-4320-ad1d-7cad3c396659-istiod-ca-cert\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.834547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.833971 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-certs\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.853553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.853347 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-credential-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.879552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.879145 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw5nq\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-kube-api-access-pw5nq\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.880857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.880811 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-envoy\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.911545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.896848 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-token\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:15.997217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:15.997183 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-podinfo\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:16.012315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:16.012285 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:16.387292 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:16.387255 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0996ac_76a8_4320_ad1d_7cad3c396659.slice/crio-594cdce0792e8e5f925e29aec52cc199042ed244e86ffbf5e99777e82a15f248 WatchSource:0}: Error finding container 594cdce0792e8e5f925e29aec52cc199042ed244e86ffbf5e99777e82a15f248: Status 404 returned error can't find the container with id 594cdce0792e8e5f925e29aec52cc199042ed244e86ffbf5e99777e82a15f248 Oct 30 01:05:16.388939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:16.388863 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx"] Oct 30 01:05:16.390009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:16.389977 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:16.390088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:16.390044 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:16.390088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:16.390073 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:17.018801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:17.018761 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" event={"ID":"5d0996ac-76a8-4320-ad1d-7cad3c396659","Type":"ContainerStarted","Data":"c5fb03af62818c99ff130317894a7288500bec8feebd059fd9f0c5862a9ce362"} Oct 30 01:05:17.019007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:17.018827 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" event={"ID":"5d0996ac-76a8-4320-ad1d-7cad3c396659","Type":"ContainerStarted","Data":"594cdce0792e8e5f925e29aec52cc199042ed244e86ffbf5e99777e82a15f248"} Oct 30 01:05:17.100810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:17.100764 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podStartSLOduration=2.100751084 podStartE2EDuration="2.100751084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:17.085220189 +0000 UTC m=+1602.028393383" watchObservedRunningTime="2026-10-30 01:05:17.100751084 +0000 UTC m=+1602.043924275" Oct 30 01:05:18.012981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.012947 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:18.019406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.019383 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:18.028564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.026675 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:18.028564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.027421 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:05:18.132709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.132621 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw"] Oct 30 01:05:18.138714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.138686 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.163310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.161799 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b483fd-openshift-default-dockercfg-rdt2m\"" Oct 30 01:05:18.182321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.181798 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw"] Oct 30 01:05:18.200593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.200562 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-podinfo\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.200839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.200649 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-data\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.200839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.200678 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.200839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.200703 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-credential-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.200839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.200760 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-envoy\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.200839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.200795 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istiod-ca-cert\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.201162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.200920 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhvw\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-kube-api-access-cvhvw\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.201162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.201004 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-certs\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.201162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.201054 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-token\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.302914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.302882 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhvw\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-kube-api-access-cvhvw\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.303106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.302964 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-certs\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.303106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303014 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-token\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.303225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303145 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-podinfo\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.303285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303219 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-data\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.303285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303248 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.303285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303274 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-credential-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.320147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303311 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-envoy\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.320147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303344 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istiod-ca-cert\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.320147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303602 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-certs\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.320147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.303981 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-data\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.320147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.304472 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-credential-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.320147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.304774 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istiod-ca-cert\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.320147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.305081 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.337440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.337163 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-podinfo\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.361020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.355240 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-envoy\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.377094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.377052 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-token\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.378640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.378611 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhvw\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-kube-api-access-cvhvw\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.482250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.481986 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:18.901563 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:18.901533 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dadcfa_aabe_4763_ac48_e8cd508e4916.slice/crio-e44569ec89574b413b3d255cdee59a9e47144ed10894a2b3ba930806b30c51a2 WatchSource:0}: Error finding container e44569ec89574b413b3d255cdee59a9e47144ed10894a2b3ba930806b30c51a2: Status 404 returned error can't find the container with id e44569ec89574b413b3d255cdee59a9e47144ed10894a2b3ba930806b30c51a2 Oct 30 01:05:18.903231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.903197 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw"] Oct 30 01:05:18.904509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.904476 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:18.904599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.904560 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:18.904599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:18.904590 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:19.065834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.065740 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" event={"ID":"c1dadcfa-aabe-4763-ac48-e8cd508e4916","Type":"ContainerStarted","Data":"e44569ec89574b413b3d255cdee59a9e47144ed10894a2b3ba930806b30c51a2"} Oct 30 01:05:19.151315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.150934 4905 patch_prober.go:29] interesting pod/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:05:19.151315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.150996 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:05:19.562419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.562383 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:05:19.605198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605165 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-envoy\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605221 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-data\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605303 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-credential-socket\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605331 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-certs\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605370 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-podinfo\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605550 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcdfb72-abd3-4976-930f-b96e5887ccef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdfb72-abd3-4976-930f-b96e5887ccef-istiod-ca-cert\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605570 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-data" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:19.605718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605675 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-credential-socket" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:19.605718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605695 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-certs" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:19.605820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605712 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-kube-api-access-s6jg7\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-kube-api-access-s6jg7\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605770 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-token\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-token\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605806 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-socket\") pod \"4bcdfb72-abd3-4976-930f-b96e5887ccef\" (UID: \"4bcdfb72-abd3-4976-930f-b96e5887ccef\") " Oct 30 01:05:19.605915 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.605886 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcdfb72-abd3-4976-930f-b96e5887ccef-istiod-ca-cert" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:19.606093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.606065 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-socket" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:19.606595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.606569 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdfb72-abd3-4976-930f-b96e5887ccef-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.606595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.606597 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.606789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.606609 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.606789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.606620 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.606789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.606631 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.608212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.608186 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-token" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:19.608306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.608257 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-podinfo" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:19.608438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.608418 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-envoy" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:19.608562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.608508 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-kube-api-access-s6jg7" pod "4bcdfb72-abd3-4976-930f-b96e5887ccef" (UID: "4bcdfb72-abd3-4976-930f-b96e5887ccef"). InnerVolumeSpecName "kube-api-access-s6jg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:19.708529 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.708468 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6jg7\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-kube-api-access-s6jg7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.708529 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.708509 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.708529 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.708536 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:19.708853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:19.708546 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcdfb72-abd3-4976-930f-b96e5887ccef-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:20.082631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.082582 4905 generic.go:356] "Generic (PLEG): container finished" podID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerID="f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a" exitCode=0 Oct 30 01:05:20.083173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.082621 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" event={"ID":"4bcdfb72-abd3-4976-930f-b96e5887ccef","Type":"ContainerDied","Data":"f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a"} Oct 30 01:05:20.083173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.082661 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" Oct 30 01:05:20.083173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.082696 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944" event={"ID":"4bcdfb72-abd3-4976-930f-b96e5887ccef","Type":"ContainerDied","Data":"0604b0605a558923a4b1e035fd3ce72718d1a4b7dc88ad58af4c03f05e1ee3fa"} Oct 30 01:05:20.083173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.082723 4905 scope.go:122] "RemoveContainer" containerID="f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a" Oct 30 01:05:20.085425 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.085387 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" event={"ID":"c1dadcfa-aabe-4763-ac48-e8cd508e4916","Type":"ContainerStarted","Data":"5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467"} Oct 30 01:05:20.097994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.097974 4905 scope.go:122] "RemoveContainer" containerID="b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680" Oct 30 01:05:20.110713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.110692 4905 scope.go:122] "RemoveContainer" containerID="f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a" Oct 30 01:05:20.111030 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:20.111009 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a\": container with ID starting with f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a not found: ID does not exist" containerID="f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a" Oct 30 01:05:20.111123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.111040 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a"} err="failed to get container status \"f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a\": rpc error: code = NotFound desc = could not find container \"f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a\": container with ID starting with f76b9eae86840cab5af9481735ef5e06c467e1e74bfffe87f456e95914c4836a not found: ID does not exist" Oct 30 01:05:20.111123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.111060 4905 scope.go:122] "RemoveContainer" containerID="b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680" Oct 30 01:05:20.111369 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:20.111345 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680\": container with ID starting with b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680 not found: ID does not exist" containerID="b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680" Oct 30 01:05:20.111461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.111376 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680"} err="failed to get container status \"b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680\": rpc error: code = NotFound desc = could not find container \"b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680\": container with ID starting with b4ddad7cd1f7309bfcd1b3936b21e0693dfa7f163352aabf0c5fe9795e94e680 not found: ID does not exist" Oct 30 01:05:20.156557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.156503 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944"] Oct 30 01:05:20.170631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.170594 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-30c405-openshift-default-64ddb7f597-sf944"] Oct 30 01:05:20.203722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.203673 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podStartSLOduration=2.203657822 podStartE2EDuration="2.203657822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:20.201098977 +0000 UTC m=+1605.144272171" watchObservedRunningTime="2026-10-30 01:05:20.203657822 +0000 UTC m=+1605.146831013" Oct 30 01:05:20.483186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.483153 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:20.489735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:20.489705 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:21.099258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.099206 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:21.100167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.100140 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:05:21.198694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.198657 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" path="/var/lib/kubelet/pods/4bcdfb72-abd3-4976-930f-b96e5887ccef/volumes" Oct 30 01:05:21.342730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.342695 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d"] Oct 30 01:05:21.343426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.343409 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" Oct 30 01:05:21.343426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.343424 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" Oct 30 01:05:21.343649 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.343567 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" Oct 30 01:05:21.343649 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.343581 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" Oct 30 01:05:21.349046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.349014 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:21.374414 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.374385 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d"] Oct 30 01:05:21.378841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.378818 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-49d9f1\"" Oct 30 01:05:21.413321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.411659 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-49d9f1-dockercfg-cx5x9\"" Oct 30 01:05:21.735262 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.735204 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q"] Oct 30 01:05:21.737293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.737265 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" Oct 30 01:05:21.737718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.737698 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcdfb72-abd3-4976-930f-b96e5887ccef" containerName="istio-proxy" Oct 30 01:05:21.744706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.744677 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:21.744986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.744962 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbhz\" (UniqueName: \"kubernetes.io/projected/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-kube-api-access-lnbhz\") pod \"payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:21.748737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.748713 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q"] Oct 30 01:05:21.748970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.748958 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:21.847470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.847433 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vx72\" (UniqueName: \"kubernetes.io/projected/b5a09346-42db-439c-bef3-96c870f851e1-kube-api-access-4vx72\") pod \"payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:21.847724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.847600 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a09346-42db-439c-bef3-96c870f851e1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:21.847724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.847640 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:21.847826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.847792 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbhz\" (UniqueName: \"kubernetes.io/projected/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-kube-api-access-lnbhz\") pod \"payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:21.848509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.848482 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:21.868721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.868693 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbhz\" (UniqueName: \"kubernetes.io/projected/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-kube-api-access-lnbhz\") pod \"payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:21.949325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.949286 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vx72\" (UniqueName: \"kubernetes.io/projected/b5a09346-42db-439c-bef3-96c870f851e1-kube-api-access-4vx72\") pod \"payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:21.949570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.949400 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a09346-42db-439c-bef3-96c870f851e1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:21.950663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.950633 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a09346-42db-439c-bef3-96c870f851e1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:21.970149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.970124 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vx72\" (UniqueName: \"kubernetes.io/projected/b5a09346-42db-439c-bef3-96c870f851e1-kube-api-access-4vx72\") pod \"payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:21.975897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:21.975874 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:22.074554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:22.071342 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:22.282347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:22.282308 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d"] Oct 30 01:05:22.391594 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:22.391558 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a09346_42db_439c_bef3_96c870f851e1.slice/crio-bfe464be243fa58681486586a897ce1fe6288a6198e9948a0ac46af3328d811e WatchSource:0}: Error finding container bfe464be243fa58681486586a897ce1fe6288a6198e9948a0ac46af3328d811e: Status 404 returned error can't find the container with id bfe464be243fa58681486586a897ce1fe6288a6198e9948a0ac46af3328d811e Oct 30 01:05:22.394303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:22.394282 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q"] Oct 30 01:05:23.118894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.118860 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" event={"ID":"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef","Type":"ContainerStarted","Data":"2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17"} Oct 30 01:05:23.119130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.118928 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:23.119130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.118955 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" event={"ID":"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef","Type":"ContainerStarted","Data":"9a5a6b887cd1d03b6de399c04ad8de43cea3359aaf9757f81daaa829cacbe394"} Oct 30 01:05:23.120700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.120670 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" event={"ID":"b5a09346-42db-439c-bef3-96c870f851e1","Type":"ContainerStarted","Data":"71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92"} Oct 30 01:05:23.120830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.120721 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" event={"ID":"b5a09346-42db-439c-bef3-96c870f851e1","Type":"ContainerStarted","Data":"bfe464be243fa58681486586a897ce1fe6288a6198e9948a0ac46af3328d811e"} Oct 30 01:05:23.120830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.120747 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:23.122468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.122444 4905 generic.go:356] "Generic (PLEG): container finished" podID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerID="2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1" exitCode=0 Oct 30 01:05:23.122618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.122529 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" event={"ID":"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56","Type":"ContainerDied","Data":"2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1"} Oct 30 01:05:23.122977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.122963 4905 scope.go:122] "RemoveContainer" containerID="2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1" Oct 30 01:05:23.124423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.124393 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:23.124568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.124459 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:23.124568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.124489 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:23.172892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.172843 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" podStartSLOduration=2.172829631 podStartE2EDuration="2.172829631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:23.166409596 +0000 UTC m=+1608.109582787" watchObservedRunningTime="2026-10-30 01:05:23.172829631 +0000 UTC m=+1608.116002822" Oct 30 01:05:23.213415 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.213374 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" podStartSLOduration=2.213362172 podStartE2EDuration="2.213362172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:23.20869869 +0000 UTC m=+1608.151871882" watchObservedRunningTime="2026-10-30 01:05:23.213362172 +0000 UTC m=+1608.156535401" Oct 30 01:05:23.892206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.892175 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:05:23.893377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:23.893349 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:05:24.178188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.178098 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" event={"ID":"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56","Type":"ContainerStarted","Data":"2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363"} Oct 30 01:05:24.609672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.609638 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:05:24.610880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.610854 4905 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 01:05:24.611026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.610908 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 01:05:24.678645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.678615 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:05:24.779020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.778981 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf"] Oct 30 01:05:24.783219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.783195 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:24.803290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.803263 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b483fd\"" Oct 30 01:05:24.803562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.803542 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b483fd-dockercfg-v4472\"" Oct 30 01:05:24.810012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.809982 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf"] Oct 30 01:05:24.813297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.813274 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps"] Oct 30 01:05:24.819116 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.819097 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:24.946592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.892551 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5459\" (UniqueName: \"kubernetes.io/projected/f49a1306-296b-4c8a-99e9-8f1698a6399d-kube-api-access-j5459\") pod \"payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:24.973541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.949751 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49a1306-296b-4c8a-99e9-8f1698a6399d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:24.973541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.949876 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzkhm\" (UniqueName: \"kubernetes.io/projected/38d0937f-af4c-40d0-add3-9361fa320b50-kube-api-access-zzkhm\") pod \"payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:24.973541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:24.949989 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d0937f-af4c-40d0-add3-9361fa320b50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:25.057717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.051581 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5459\" (UniqueName: \"kubernetes.io/projected/f49a1306-296b-4c8a-99e9-8f1698a6399d-kube-api-access-j5459\") pod \"payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:25.057717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.051749 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49a1306-296b-4c8a-99e9-8f1698a6399d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:25.057717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.051830 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzkhm\" (UniqueName: \"kubernetes.io/projected/38d0937f-af4c-40d0-add3-9361fa320b50-kube-api-access-zzkhm\") pod \"payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:25.057717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.051921 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d0937f-af4c-40d0-add3-9361fa320b50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:25.057717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.052893 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d0937f-af4c-40d0-add3-9361fa320b50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:25.076487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.061970 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49a1306-296b-4c8a-99e9-8f1698a6399d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:25.192140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.192112 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5459\" (UniqueName: \"kubernetes.io/projected/f49a1306-296b-4c8a-99e9-8f1698a6399d-kube-api-access-j5459\") pod \"payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:25.192872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.192844 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzkhm\" (UniqueName: \"kubernetes.io/projected/38d0937f-af4c-40d0-add3-9361fa320b50-kube-api-access-zzkhm\") pod \"payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:25.200890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.200824 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps"] Oct 30 01:05:25.399325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.399292 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:25.494086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.493322 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:25.610043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.610011 4905 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 01:05:25.610163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.610075 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 01:05:25.754350 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:25.748710 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d0937f_af4c_40d0_add3_9361fa320b50.slice/crio-aa1fbc884bc89c94e83f864a4f9c9cbaebd846dc717e540f41221c38a02aabd4 WatchSource:0}: Error finding container aa1fbc884bc89c94e83f864a4f9c9cbaebd846dc717e540f41221c38a02aabd4: Status 404 returned error can't find the container with id aa1fbc884bc89c94e83f864a4f9c9cbaebd846dc717e540f41221c38a02aabd4 Oct 30 01:05:25.758360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.758335 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf"] Oct 30 01:05:25.860157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:25.859786 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps"] Oct 30 01:05:26.310554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.310063 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" event={"ID":"38d0937f-af4c-40d0-add3-9361fa320b50","Type":"ContainerStarted","Data":"ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11"} Oct 30 01:05:26.310554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.310187 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" event={"ID":"38d0937f-af4c-40d0-add3-9361fa320b50","Type":"ContainerStarted","Data":"aa1fbc884bc89c94e83f864a4f9c9cbaebd846dc717e540f41221c38a02aabd4"} Oct 30 01:05:26.311389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.311367 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:26.324320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.324272 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" event={"ID":"f49a1306-296b-4c8a-99e9-8f1698a6399d","Type":"ContainerStarted","Data":"53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce"} Oct 30 01:05:26.324454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.324340 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" event={"ID":"f49a1306-296b-4c8a-99e9-8f1698a6399d","Type":"ContainerStarted","Data":"f219fb209c05a223335eb98f21f69e5af36a6fb3f2f3e672f9572bf587cf8fee"} Oct 30 01:05:26.324454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.324386 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:26.369101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.369056 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" podStartSLOduration=2.369042761 podStartE2EDuration="2.369042761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:26.366553162 +0000 UTC m=+1611.309726355" watchObservedRunningTime="2026-10-30 01:05:26.369042761 +0000 UTC m=+1611.312215944" Oct 30 01:05:26.415438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.415386 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" podStartSLOduration=2.41537045 podStartE2EDuration="2.41537045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:26.409224683 +0000 UTC m=+1611.352397876" watchObservedRunningTime="2026-10-30 01:05:26.41537045 +0000 UTC m=+1611.358543645" Oct 30 01:05:26.610010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.609922 4905 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 01:05:26.610281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:26.610253 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 01:05:27.609555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:27.609509 4905 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 01:05:27.609950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:27.609581 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 01:05:28.610458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:28.610432 4905 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 01:05:28.610985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:28.610597 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 01:05:28.821307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:28.821274 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr"] Oct 30 01:05:28.936613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:28.936031 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee0869-h2sxr"] Oct 30 01:05:29.259542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:29.258640 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338b099e-30c4-4f31-a6c8-8be499d56267" path="/var/lib/kubelet/pods/338b099e-30c4-4f31-a6c8-8be499d56267/volumes" Oct 30 01:05:29.609667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:29.609594 4905 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 01:05:29.609871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:29.609847 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 01:05:30.614121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:30.614073 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:05:30.615365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:30.615340 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:05:34.033510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:34.033467 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz"] Oct 30 01:05:34.046198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:34.046151 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99549ef3-srgdz"] Oct 30 01:05:34.051738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:34.051697 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx"] Oct 30 01:05:34.056858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:34.056830 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb2911f-gvjpx"] Oct 30 01:05:34.180937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:34.180912 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:05:34.181118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:34.180984 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:05:35.197351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:35.197317 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bf8159-6bb7-4791-aa29-7e59dd281033" path="/var/lib/kubelet/pods/19bf8159-6bb7-4791-aa29-7e59dd281033/volumes" Oct 30 01:05:35.198001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:35.197983 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8fadf7-9f08-4979-9aee-11185d539e50" path="/var/lib/kubelet/pods/3d8fadf7-9f08-4979-9aee-11185d539e50/volumes" Oct 30 01:05:35.894337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:35.894223 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk"] Oct 30 01:05:35.894937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:35.894892 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" containerID="cri-o://4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d" gracePeriod=30 Oct 30 01:05:37.129586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.129551 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr"] Oct 30 01:05:37.137657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.137634 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.163770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.163654 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8e8ad1-openshift-default-dockercfg-f8vxc\"" Oct 30 01:05:37.172105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.172084 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr"] Oct 30 01:05:37.206390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206357 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-token\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206395 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-envoy\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206471 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-data\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206563 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-podinfo\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206589 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-certs\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206618 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-credential-socket\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206639 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-socket\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206689 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ff57b2-7fb3-40bb-a720-7c35964885f2-istiod-ca-cert\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.206852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.206769 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkk7\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-kube-api-access-zrkk7\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.307905 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkk7\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-kube-api-access-zrkk7\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308042 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-token\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308083 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-envoy\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308147 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-data\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308221 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-podinfo\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308263 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-certs\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308323 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-credential-socket\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308359 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-socket\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.308580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.308495 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ff57b2-7fb3-40bb-a720-7c35964885f2-istiod-ca-cert\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.310595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.309984 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-certs\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.310595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.310030 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-socket\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.310595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.310289 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-credential-socket\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.311040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.311006 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-data\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.312597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.312505 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ff57b2-7fb3-40bb-a720-7c35964885f2-istiod-ca-cert\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.315781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.315761 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-envoy\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.473595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.448609 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-podinfo\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.638296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.638014 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-token\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.643836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.641461 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkk7\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-kube-api-access-zrkk7\") pod \"e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:37.645328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.644248 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:05:37.781565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:37.774078 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:38.262547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.255544 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr"] Oct 30 01:05:38.279902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.279863 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:38.280029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.279943 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:38.280029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.279977 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:38.401382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.401355 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:05:38.660393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.658980 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" event={"ID":"54ff57b2-7fb3-40bb-a720-7c35964885f2","Type":"ContainerStarted","Data":"e4d04365958fa41c2c086106555bc822b2e86567a3850358a525a1dd469cc708"} Oct 30 01:05:38.660393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.659047 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" event={"ID":"54ff57b2-7fb3-40bb-a720-7c35964885f2","Type":"ContainerStarted","Data":"2bcc00bd1fe7984b9edbeeba75d420e18ccd0f0c856f7a95f9ba516085d960f4"} Oct 30 01:05:38.706308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.706259 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podStartSLOduration=1.706245457 podStartE2EDuration="1.706245457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:38.705157703 +0000 UTC m=+1623.648330897" watchObservedRunningTime="2026-10-30 01:05:38.706245457 +0000 UTC m=+1623.649418650" Oct 30 01:05:38.779920 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:38.779886 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:39.341774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.341746 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": read tcp 10.132.0.2:57518->10.132.0.205:15021: read: connection reset by peer" start-of-body= Oct 30 01:05:39.342230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.341826 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": read tcp 10.132.0.2:57518->10.132.0.205:15021: read: connection reset by peer" Oct 30 01:05:39.669772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.669677 4905 generic.go:356] "Generic (PLEG): container finished" podID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerID="0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25" exitCode=0 Oct 30 01:05:39.669772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.669753 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" event={"ID":"b272f295-7d0a-49d2-af8d-4970059d17a7","Type":"ContainerDied","Data":"0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25"} Oct 30 01:05:39.670507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.670485 4905 scope.go:122] "RemoveContainer" containerID="0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25" Oct 30 01:05:39.672311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.672278 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:39.672430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.672360 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:39.672430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.672412 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:39.733503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.733457 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs"] Oct 30 01:05:39.748220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.748042 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.768112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.766799 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bd9cc5-serving-cert\"" Oct 30 01:05:39.772497 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.772471 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs"] Oct 30 01:05:39.788531 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.788484 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:39.833708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.833665 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcxm\" (UniqueName: \"kubernetes.io/projected/7a99d858-b126-4780-9d53-2004398b2da9-kube-api-access-9pcxm\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.833878 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.833784 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-maas-api-tls\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.833954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.833935 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-metrics-certs\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.834015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.833995 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a99d858-b126-4780-9d53-2004398b2da9-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.935836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.935743 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-maas-api-tls\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.940549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.936495 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-metrics-certs\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.940549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.937023 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a99d858-b126-4780-9d53-2004398b2da9-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.953543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.943084 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a99d858-b126-4780-9d53-2004398b2da9-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:39.984594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:39.983872 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-maas-api-tls\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:40.004549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.001250 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-metrics-certs\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:40.004549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.002109 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcxm\" (UniqueName: \"kubernetes.io/projected/7a99d858-b126-4780-9d53-2004398b2da9-kube-api-access-9pcxm\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:40.039542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.039299 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcxm\" (UniqueName: \"kubernetes.io/projected/7a99d858-b126-4780-9d53-2004398b2da9-kube-api-access-9pcxm\") pod \"maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:40.095101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.095056 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:40.428625 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:40.428591 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a99d858_b126_4780_9d53_2004398b2da9.slice/crio-d86bd1fdcd9c35011d9ab59208c6cd5ecd90c9503acc326fa8a2e6353c8c3a67 WatchSource:0}: Error finding container d86bd1fdcd9c35011d9ab59208c6cd5ecd90c9503acc326fa8a2e6353c8c3a67: Status 404 returned error can't find the container with id d86bd1fdcd9c35011d9ab59208c6cd5ecd90c9503acc326fa8a2e6353c8c3a67 Oct 30 01:05:40.429676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.429651 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs"] Oct 30 01:05:40.690806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.690713 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" event={"ID":"7a99d858-b126-4780-9d53-2004398b2da9","Type":"ContainerStarted","Data":"d86bd1fdcd9c35011d9ab59208c6cd5ecd90c9503acc326fa8a2e6353c8c3a67"} Oct 30 01:05:40.692868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.692841 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" event={"ID":"b272f295-7d0a-49d2-af8d-4970059d17a7","Type":"ContainerStarted","Data":"6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343"} Oct 30 01:05:40.693065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.693049 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:40.694242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:40.694223 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:05:41.367579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.364674 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 01:05:41.367579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.364737 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 01:05:41.367579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.366494 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:05:41.611864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.611393 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:05:41.692821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.692731 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-certs\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.692846 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-envoy\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.692970 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd40be8-c62e-4831-8fe8-075ae7835375-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd40be8-c62e-4831-8fe8-075ae7835375-istiod-ca-cert\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.693052 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-socket\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.693116 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-podinfo\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.693163 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-credential-socket\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.693225 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-kube-api-access-vsfrx\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-kube-api-access-vsfrx\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.693339 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-token\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-token\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.693455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.693436 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-data\") pod \"5cd40be8-c62e-4831-8fe8-075ae7835375\" (UID: \"5cd40be8-c62e-4831-8fe8-075ae7835375\") " Oct 30 01:05:41.694301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.694271 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-socket" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:41.694751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.694727 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-credential-socket" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:41.695081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.695057 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd40be8-c62e-4831-8fe8-075ae7835375-istiod-ca-cert" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:41.695368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.695344 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-certs" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:41.697203 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.697167 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-envoy" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:41.701678 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.701598 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-data" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:41.705455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.705417 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-token" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:41.707412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.707374 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-podinfo" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:41.714662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714636 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.714859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714668 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.714859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714681 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.714859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714697 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd40be8-c62e-4831-8fe8-075ae7835375-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.714859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714710 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.714859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714724 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.714859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714740 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd40be8-c62e-4831-8fe8-075ae7835375-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.714859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.714753 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.716329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.716290 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-kube-api-access-vsfrx" pod "5cd40be8-c62e-4831-8fe8-075ae7835375" (UID: "5cd40be8-c62e-4831-8fe8-075ae7835375"). InnerVolumeSpecName "kube-api-access-vsfrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:41.718375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.718337 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" event={"ID":"7a99d858-b126-4780-9d53-2004398b2da9","Type":"ContainerStarted","Data":"0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403"} Oct 30 01:05:41.718676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.718660 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:41.721488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.721463 4905 generic.go:356] "Generic (PLEG): container finished" podID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerID="4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d" exitCode=0 Oct 30 01:05:41.722651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.722633 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" Oct 30 01:05:41.750751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.745982 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" event={"ID":"5cd40be8-c62e-4831-8fe8-075ae7835375","Type":"ContainerDied","Data":"4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d"} Oct 30 01:05:41.750751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.746079 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk" event={"ID":"5cd40be8-c62e-4831-8fe8-075ae7835375","Type":"ContainerDied","Data":"d36c90ead07ad818e2033c7caf961606d4cc7b339e655f742e0fa84ad3b3aa72"} Oct 30 01:05:41.750751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.746115 4905 scope.go:122] "RemoveContainer" containerID="4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d" Oct 30 01:05:41.819126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.819085 4905 scope.go:122] "RemoveContainer" containerID="39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0" Oct 30 01:05:41.819126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.819100 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsfrx\" (UniqueName: \"kubernetes.io/projected/5cd40be8-c62e-4831-8fe8-075ae7835375-kube-api-access-vsfrx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:41.873740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.867455 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" podStartSLOduration=2.48251497 podStartE2EDuration="2.867437806s" totalImagesPullingTime="384.922836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:41.801476383 +0000 UTC m=+1626.744649577" watchObservedRunningTime="2026-10-30 01:05:41.867437806 +0000 UTC m=+1626.810611001" Oct 30 01:05:41.880880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.880855 4905 scope.go:122] "RemoveContainer" containerID="4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d" Oct 30 01:05:41.881475 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:41.881450 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d\": container with ID starting with 4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d not found: ID does not exist" containerID="4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d" Oct 30 01:05:41.882002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.881580 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d"} err="failed to get container status \"4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d\": rpc error: code = NotFound desc = could not find container \"4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d\": container with ID starting with 4631c54d2f4cba04f1ec5d8e05d164c49ded748619cf4ea821d3393096fe7d2d not found: ID does not exist" Oct 30 01:05:41.882002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.881607 4905 scope.go:122] "RemoveContainer" containerID="39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0" Oct 30 01:05:41.882002 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:41.881958 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0\": container with ID starting with 39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0 not found: ID does not exist" containerID="39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0" Oct 30 01:05:41.882198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.881993 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0"} err="failed to get container status \"39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0\": rpc error: code = NotFound desc = could not find container \"39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0\": container with ID starting with 39d8b430a8e0b775b1f15c24c90600bf6b435c1c9f1d99e7c329a5f283c618d0 not found: ID does not exist" Oct 30 01:05:41.885919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.885896 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk"] Oct 30 01:05:41.891257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:41.891231 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ee0869-openshift-default-7d576b8d7c-kl7sk"] Oct 30 01:05:42.021423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.021392 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s"] Oct 30 01:05:42.022286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.022264 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" Oct 30 01:05:42.022286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.022284 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" Oct 30 01:05:42.022466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.022425 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" Oct 30 01:05:42.022466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.022437 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" Oct 30 01:05:42.058670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.053177 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.075356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.075325 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0680fa-openshift-default-dockercfg-gxjnz\"" Oct 30 01:05:42.076352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.076328 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s"] Oct 30 01:05:42.123742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.123707 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-certs\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.123992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.123972 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-envoy\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.124177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.124162 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-token\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.124310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.124289 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk85h\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-kube-api-access-mk85h\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.124406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.124389 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f3e0d6-a86c-4527-ab11-e6732c123c73-istiod-ca-cert\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.124488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.124464 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.148543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.128766 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-podinfo\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.148543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.128960 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-data\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.148543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.128998 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-credential-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.255668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.230508 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-credential-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.256081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.256049 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-credential-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.256321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.256300 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-certs\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.256442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.256427 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-envoy\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.256683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.256666 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-token\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.256819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.256805 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk85h\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-kube-api-access-mk85h\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.256973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.256958 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f3e0d6-a86c-4527-ab11-e6732c123c73-istiod-ca-cert\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.257070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.257058 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.257201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.257188 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-podinfo\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.257496 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.257477 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-data\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.258024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.258001 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-data\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.258153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.258106 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.258624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.258598 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f3e0d6-a86c-4527-ab11-e6732c123c73-istiod-ca-cert\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.259421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.259382 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-certs\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.260668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.260576 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-envoy\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.335666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.335577 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk85h\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-kube-api-access-mk85h\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.346846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.346719 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-podinfo\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.348725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.348697 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 01:05:42.348970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.348937 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 01:05:42.363127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.361251 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-token\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.397541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.393817 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:42.682682 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:42.682645 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:05:42.892161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.892069 4905 scope.go:122] "RemoveContainer" containerID="ea48301d88dc997ec6c1707d73d013e15bc5586b6bffbbee4de92d3ad0c6eaa2" Oct 30 01:05:42.910549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.910499 4905 scope.go:122] "RemoveContainer" containerID="da4eed6ce66ada5f1eff37b8cbf28eaa671f8d8cabd3c1020c7526585458a782" Oct 30 01:05:42.963671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.941101 4905 scope.go:122] "RemoveContainer" containerID="fa56d903ced590e176179f871fbc079ed501d87336e4e46ac1eb96d2a270e76c" Oct 30 01:05:42.977591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:42.977498 4905 scope.go:122] "RemoveContainer" containerID="5cfa791c345461f8ae4c6178d82303af9df1fb65a9bf7cac6d2dfa8d177f2d7c" Oct 30 01:05:43.064608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.064577 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s"] Oct 30 01:05:43.072350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.072307 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6"] Oct 30 01:05:43.080195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.080161 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:43.080360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.080343 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:43.080480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.080465 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:43.084611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.084587 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d49641-br5j6"] Oct 30 01:05:43.194251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.194225 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" path="/var/lib/kubelet/pods/5cd40be8-c62e-4831-8fe8-075ae7835375/volumes" Oct 30 01:05:43.195004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.194979 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7131451d-9f5d-48b9-bf05-11d18be41535" path="/var/lib/kubelet/pods/7131451d-9f5d-48b9-bf05-11d18be41535/volumes" Oct 30 01:05:43.349302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.348861 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 01:05:43.349302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.348920 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 01:05:43.799411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.799375 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" event={"ID":"74f3e0d6-a86c-4527-ab11-e6732c123c73","Type":"ContainerStarted","Data":"9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8"} Oct 30 01:05:43.799831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.799449 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" event={"ID":"74f3e0d6-a86c-4527-ab11-e6732c123c73","Type":"ContainerStarted","Data":"9156e44bdaa06bac462741da9d8c84f82f2917bf466c40b7c7717588a103c5fd"} Oct 30 01:05:43.867918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.867880 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv"] Oct 30 01:05:43.869413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.869384 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" Oct 30 01:05:43.869413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.869413 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd40be8-c62e-4831-8fe8-075ae7835375" containerName="istio-proxy" Oct 30 01:05:43.874665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.874606 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podStartSLOduration=2.874588783 podStartE2EDuration="2.874588783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:43.86738693 +0000 UTC m=+1628.810560124" watchObservedRunningTime="2026-10-30 01:05:43.874588783 +0000 UTC m=+1628.817761979" Oct 30 01:05:43.883081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.881881 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv"] Oct 30 01:05:43.883081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.882071 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:43.899405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.899374 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv"] Oct 30 01:05:43.904682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.904656 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8e8ad1\"" Oct 30 01:05:43.906019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.905651 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:43.910912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.910881 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8e8ad1-dockercfg-m52tf\"" Oct 30 01:05:43.992170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:43.985721 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv"] Oct 30 01:05:44.078470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.077631 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692c5ba6-0bf9-4f73-909f-28a5fc55a572-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:44.078470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.077741 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvnh\" (UniqueName: \"kubernetes.io/projected/692c5ba6-0bf9-4f73-909f-28a5fc55a572-kube-api-access-wjvnh\") pod \"payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:44.078470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.077855 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l94w\" (UniqueName: \"kubernetes.io/projected/e6ed45f5-c974-4e73-a92c-60189fe4dc03-kube-api-access-7l94w\") pod \"payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:44.078470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.078027 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ed45f5-c974-4e73-a92c-60189fe4dc03-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:44.228645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.197830 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ed45f5-c974-4e73-a92c-60189fe4dc03-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:44.228645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.197999 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692c5ba6-0bf9-4f73-909f-28a5fc55a572-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:44.228645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.198077 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvnh\" (UniqueName: \"kubernetes.io/projected/692c5ba6-0bf9-4f73-909f-28a5fc55a572-kube-api-access-wjvnh\") pod \"payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:44.228645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.198182 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l94w\" (UniqueName: \"kubernetes.io/projected/e6ed45f5-c974-4e73-a92c-60189fe4dc03-kube-api-access-7l94w\") pod \"payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:44.228645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.199442 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ed45f5-c974-4e73-a92c-60189fe4dc03-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:44.251032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.250976 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692c5ba6-0bf9-4f73-909f-28a5fc55a572-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:44.313812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.313180 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvnh\" (UniqueName: \"kubernetes.io/projected/692c5ba6-0bf9-4f73-909f-28a5fc55a572-kube-api-access-wjvnh\") pod \"payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:44.390751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.362032 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 01:05:44.390751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.362099 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 01:05:44.390751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.368707 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l94w\" (UniqueName: \"kubernetes.io/projected/e6ed45f5-c974-4e73-a92c-60189fe4dc03-kube-api-access-7l94w\") pod \"payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:44.395334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.395266 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:44.402509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.402487 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:44.509480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.509450 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:44.631605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.630688 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:44.814482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.814429 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:44.816012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.815987 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:05:44.924482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.924449 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn"] Oct 30 01:05:44.948539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:44.947825 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" containerID="cri-o://6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9" gracePeriod=30 Oct 30 01:05:45.427681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.389134 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 01:05:45.427681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.389198 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 01:05:45.652991 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.652883 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv"] Oct 30 01:05:45.653329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.653300 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv"] Oct 30 01:05:45.839684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.838454 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" event={"ID":"692c5ba6-0bf9-4f73-909f-28a5fc55a572","Type":"ContainerStarted","Data":"44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6"} Oct 30 01:05:45.839684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.838537 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" event={"ID":"692c5ba6-0bf9-4f73-909f-28a5fc55a572","Type":"ContainerStarted","Data":"71042e567187423771cb78e32adda6374f19ca83392f7fa265f8c2da9e4c613e"} Oct 30 01:05:45.839684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.839632 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:45.846735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.846704 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" event={"ID":"e6ed45f5-c974-4e73-a92c-60189fe4dc03","Type":"ContainerStarted","Data":"d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db"} Oct 30 01:05:45.847051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.847033 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:45.847205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.847188 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" event={"ID":"e6ed45f5-c974-4e73-a92c-60189fe4dc03","Type":"ContainerStarted","Data":"e6dea4b9ba288cf910b829633c3ad2b393ad6ba5bab6f3c957723daa2917c355"} Oct 30 01:05:45.888471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.888409 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" podStartSLOduration=2.888397163 podStartE2EDuration="2.888397163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:45.886641902 +0000 UTC m=+1630.829815095" watchObservedRunningTime="2026-10-30 01:05:45.888397163 +0000 UTC m=+1630.831570356" Oct 30 01:05:45.944839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:45.944776 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" podStartSLOduration=2.944753652 podStartE2EDuration="2.944753652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:45.937066203 +0000 UTC m=+1630.880239393" watchObservedRunningTime="2026-10-30 01:05:45.944753652 +0000 UTC m=+1630.887926846" Oct 30 01:05:46.351844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:46.351801 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 01:05:46.352034 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:46.351868 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 01:05:47.485994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:47.485949 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:05:47.486510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:47.486416 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:05:47.487230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:47.487210 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:05:47.796444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:47.796355 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:05:48.253249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.253217 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp"] Oct 30 01:05:48.260081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.260053 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.281027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.281001 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0680fa\"" Oct 30 01:05:48.303646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.303607 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp"] Oct 30 01:05:48.367898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.367846 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0680fa-dockercfg-vnp79\"" Oct 30 01:05:48.429554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.415195 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc3397b-4c60-40f4-b0d0-950829147631-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.430682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.430647 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54r4x\" (UniqueName: \"kubernetes.io/projected/9bc3397b-4c60-40f4-b0d0-950829147631-kube-api-access-54r4x\") pod \"payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.431294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.428584 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7"] Oct 30 01:05:48.491953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.491915 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:48.496799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.494719 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7"] Oct 30 01:05:48.665816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.665786 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc3397b-4c60-40f4-b0d0-950829147631-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.670389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.666882 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54r4x\" (UniqueName: \"kubernetes.io/projected/9bc3397b-4c60-40f4-b0d0-950829147631-kube-api-access-54r4x\") pod \"payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.670389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.667500 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc3397b-4c60-40f4-b0d0-950829147631-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.704235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.704200 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54r4x\" (UniqueName: \"kubernetes.io/projected/9bc3397b-4c60-40f4-b0d0-950829147631-kube-api-access-54r4x\") pod \"payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.768338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.768295 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7zh\" (UniqueName: \"kubernetes.io/projected/ed618312-e21b-412e-87f1-484cad8a0974-kube-api-access-nv7zh\") pod \"payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:48.768504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.768455 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed618312-e21b-412e-87f1-484cad8a0974-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:48.869861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.869772 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7zh\" (UniqueName: \"kubernetes.io/projected/ed618312-e21b-412e-87f1-484cad8a0974-kube-api-access-nv7zh\") pod \"payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:48.870040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.869906 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed618312-e21b-412e-87f1-484cad8a0974-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:48.870887 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.870863 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed618312-e21b-412e-87f1-484cad8a0974-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:48.878456 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.878419 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:48.891874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.891844 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7zh\" (UniqueName: \"kubernetes.io/projected/ed618312-e21b-412e-87f1-484cad8a0974-kube-api-access-nv7zh\") pod \"payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:48.961587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:48.961307 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:49.216907 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:49.216874 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc3397b_4c60_40f4_b0d0_950829147631.slice/crio-00ec752574df1bc122647a8cc46da6aa5115c906bc31e5da8cffeb72b1d65b31 WatchSource:0}: Error finding container 00ec752574df1bc122647a8cc46da6aa5115c906bc31e5da8cffeb72b1d65b31: Status 404 returned error can't find the container with id 00ec752574df1bc122647a8cc46da6aa5115c906bc31e5da8cffeb72b1d65b31 Oct 30 01:05:49.217811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.217787 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp"] Oct 30 01:05:49.295858 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:49.295829 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded618312_e21b_412e_87f1_484cad8a0974.slice/crio-96ea3ec82a09df8b06dbe87367ca8802b08b5bd20fe8c774fce6ffb57182b072 WatchSource:0}: Error finding container 96ea3ec82a09df8b06dbe87367ca8802b08b5bd20fe8c774fce6ffb57182b072: Status 404 returned error can't find the container with id 96ea3ec82a09df8b06dbe87367ca8802b08b5bd20fe8c774fce6ffb57182b072 Oct 30 01:05:49.297251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.297224 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7"] Oct 30 01:05:49.781314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.781275 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s"] Oct 30 01:05:49.787730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.787689 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.814752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.814698 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s"] Oct 30 01:05:49.822765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.822732 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-286d36-openshift-default-dockercfg-qmrk9\"" Oct 30 01:05:49.890399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890360 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-certs\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.890399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890406 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.890694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890542 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-credential-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.890750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890694 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8kb\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-kube-api-access-qr8kb\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.890810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890760 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef45997-65ed-4532-8fb2-ce882d7ec527-istiod-ca-cert\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.890864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890810 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-podinfo\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.890918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890880 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-envoy\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.890918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.890901 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-token\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.891076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.891058 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-data\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:49.961051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.961008 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" event={"ID":"9bc3397b-4c60-40f4-b0d0-950829147631","Type":"ContainerStarted","Data":"876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b"} Oct 30 01:05:49.961273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.961078 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" event={"ID":"9bc3397b-4c60-40f4-b0d0-950829147631","Type":"ContainerStarted","Data":"00ec752574df1bc122647a8cc46da6aa5115c906bc31e5da8cffeb72b1d65b31"} Oct 30 01:05:49.961273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.961111 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:05:49.963847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.963815 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" event={"ID":"ed618312-e21b-412e-87f1-484cad8a0974","Type":"ContainerStarted","Data":"ae378a280d1a871b8d3feb42e75df64036d95a7c626c71856e6e94e467d5154a"} Oct 30 01:05:49.964016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.963866 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" event={"ID":"ed618312-e21b-412e-87f1-484cad8a0974","Type":"ContainerStarted","Data":"96ea3ec82a09df8b06dbe87367ca8802b08b5bd20fe8c774fce6ffb57182b072"} Oct 30 01:05:49.964603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.964586 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:05:49.995937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:49.995900 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.005198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.005170 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-credential-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.006288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.006258 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8kb\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-kube-api-access-qr8kb\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.032340 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.032255 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef45997-65ed-4532-8fb2-ce882d7ec527-istiod-ca-cert\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.033289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.033267 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-podinfo\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.033984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.033961 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-envoy\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.034461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.034441 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-token\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.035065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.035045 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-data\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.035998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.035979 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-certs\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.036538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.033180 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef45997-65ed-4532-8fb2-ce882d7ec527-istiod-ca-cert\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.036688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.005031 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.036792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.035787 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-data\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.036890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.006049 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-credential-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.036979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.036336 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-certs\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.060980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.060943 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-envoy\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.084547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.078769 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8kb\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-kube-api-access-qr8kb\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.139163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.139128 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-token\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.141495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.141464 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-podinfo\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.383629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.382989 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" podStartSLOduration=2.382968257 podStartE2EDuration="2.382968257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:50.381899536 +0000 UTC m=+1635.325072731" watchObservedRunningTime="2026-10-30 01:05:50.382968257 +0000 UTC m=+1635.326141452" Oct 30 01:05:50.384880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.384824 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" podStartSLOduration=2.38480893 podStartE2EDuration="2.38480893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:50.082050032 +0000 UTC m=+1635.025223222" watchObservedRunningTime="2026-10-30 01:05:50.38480893 +0000 UTC m=+1635.327982123" Oct 30 01:05:50.452968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.421679 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:50.816862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.816823 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:05:50.893124 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.893090 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s"] Oct 30 01:05:50.898575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.898534 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:50.898689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.898633 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:50.898689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.898681 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:50.976700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.976663 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" event={"ID":"7ef45997-65ed-4532-8fb2-ce882d7ec527","Type":"ContainerStarted","Data":"7a9e13b0593774960008202a6cffa4e701df463c90cc38c19e4de7abfd97ce69"} Oct 30 01:05:50.978672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.978644 4905 generic.go:356] "Generic (PLEG): container finished" podID="34b8803c-bbf8-434b-91de-024c924643cc" containerID="6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9" exitCode=0 Oct 30 01:05:50.978824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.978728 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" Oct 30 01:05:50.978824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.978758 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" event={"ID":"34b8803c-bbf8-434b-91de-024c924643cc","Type":"ContainerDied","Data":"6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9"} Oct 30 01:05:50.979072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.978823 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn" event={"ID":"34b8803c-bbf8-434b-91de-024c924643cc","Type":"ContainerDied","Data":"939b3ff2391e837a418c8300c0a43b17e09004c706d0a19ab3fa963a13443787"} Oct 30 01:05:50.979072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.978848 4905 scope.go:122] "RemoveContainer" containerID="6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9" Oct 30 01:05:50.987617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987592 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-data\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.987750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987637 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-socket\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.987750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987728 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-credential-socket\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.987879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987773 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-certs\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.987940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987874 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-data" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:50.987940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987880 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-envoy\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.987940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987910 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-socket" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:50.988049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.987981 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b8803c-bbf8-434b-91de-024c924643cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b8803c-bbf8-434b-91de-024c924643cc-istiod-ca-cert\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.988100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988091 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b8803c-bbf8-434b-91de-024c924643cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b8803c-bbf8-434b-91de-024c924643cc-istio-podinfo\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.988165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988134 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-credential-socket" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:50.988200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988163 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-istio-token\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-istio-token\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.988308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988287 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-kube-api-access-2bkrh\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-kube-api-access-2bkrh\") pod \"34b8803c-bbf8-434b-91de-024c924643cc\" (UID: \"34b8803c-bbf8-434b-91de-024c924643cc\") " Oct 30 01:05:50.988399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988377 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-certs" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:50.988454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988424 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b8803c-bbf8-434b-91de-024c924643cc-istiod-ca-cert" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:50.988937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988915 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:50.989249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988941 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:50.989249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988957 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:50.989249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988973 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:50.989249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.988986 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b8803c-bbf8-434b-91de-024c924643cc-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:50.990605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.990509 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-envoy" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:50.990923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.990808 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b8803c-bbf8-434b-91de-024c924643cc-istio-podinfo" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:50.990923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.990896 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-istio-token" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:50.990923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.990914 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-kube-api-access-2bkrh" pod "34b8803c-bbf8-434b-91de-024c924643cc" (UID: "34b8803c-bbf8-434b-91de-024c924643cc"). InnerVolumeSpecName "kube-api-access-2bkrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:50.992367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:50.992351 4905 scope.go:122] "RemoveContainer" containerID="af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d" Oct 30 01:05:51.003422 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.003402 4905 scope.go:122] "RemoveContainer" containerID="6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9" Oct 30 01:05:51.003727 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:51.003707 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9\": container with ID starting with 6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9 not found: ID does not exist" containerID="6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9" Oct 30 01:05:51.003828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.003731 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9"} err="failed to get container status \"6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9\": rpc error: code = NotFound desc = could not find container \"6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9\": container with ID starting with 6cbdca3104ac37d681aebef162513fb4801388f0c027149de64ad4d369eb65b9 not found: ID does not exist" Oct 30 01:05:51.003828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.003747 4905 scope.go:122] "RemoveContainer" containerID="af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d" Oct 30 01:05:51.003994 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:05:51.003978 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d\": container with ID starting with af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d not found: ID does not exist" containerID="af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d" Oct 30 01:05:51.004051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.003998 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d"} err="failed to get container status \"af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d\": rpc error: code = NotFound desc = could not find container \"af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d\": container with ID starting with af894a9d6e0f6b8c9b7a4f26da3967e92d9185a0ef8c8d5c80d374ef09d0d08d not found: ID does not exist" Oct 30 01:05:51.090852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.090813 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:51.090852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.090845 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bkrh\" (UniqueName: \"kubernetes.io/projected/34b8803c-bbf8-434b-91de-024c924643cc-kube-api-access-2bkrh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:51.091118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.090861 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b8803c-bbf8-434b-91de-024c924643cc-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:51.091118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.090878 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b8803c-bbf8-434b-91de-024c924643cc-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:05:51.325379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.325346 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn"] Oct 30 01:05:51.332295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.332267 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-mljpn"] Oct 30 01:05:51.990354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:51.990318 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" event={"ID":"7ef45997-65ed-4532-8fb2-ce882d7ec527","Type":"ContainerStarted","Data":"3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068"} Oct 30 01:05:52.037228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:52.037168 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podStartSLOduration=3.037146417 podStartE2EDuration="3.037146417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:52.035563212 +0000 UTC m=+1636.978736406" watchObservedRunningTime="2026-10-30 01:05:52.037146417 +0000 UTC m=+1636.980319610" Oct 30 01:05:52.429047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:52.428965 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:52.436164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:52.436140 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:53.005207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:53.005175 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:53.006112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:53.006085 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:05:53.295056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:53.294974 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b8803c-bbf8-434b-91de-024c924643cc" path="/var/lib/kubelet/pods/34b8803c-bbf8-434b-91de-024c924643cc/volumes" Oct 30 01:05:56.309101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.309064 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc"] Oct 30 01:05:56.309944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.309928 4905 cpu_manager.go:414] "containerMap: removing container" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" Oct 30 01:05:56.309996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.309945 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" Oct 30 01:05:56.309996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.309963 4905 cpu_manager.go:414] "containerMap: removing container" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" Oct 30 01:05:56.309996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.309968 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" Oct 30 01:05:56.310191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.310179 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" Oct 30 01:05:56.310230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.310194 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b8803c-bbf8-434b-91de-024c924643cc" containerName="istio-proxy" Oct 30 01:05:56.314227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.314205 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:56.334088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.334057 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc"] Oct 30 01:05:56.342162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.342131 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-286d36\"" Oct 30 01:05:56.374193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.374154 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275b80bd-da5b-40b2-8f10-3fddaf2aebab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-286d36-54c688684b-z64mc\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:56.374387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.374354 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6qd\" (UniqueName: \"kubernetes.io/projected/275b80bd-da5b-40b2-8f10-3fddaf2aebab-kube-api-access-zk6qd\") pod \"payload-processing-e2e-inf-a-286d36-54c688684b-z64mc\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:56.787631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.787587 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275b80bd-da5b-40b2-8f10-3fddaf2aebab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-286d36-54c688684b-z64mc\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:56.787808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.787790 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6qd\" (UniqueName: \"kubernetes.io/projected/275b80bd-da5b-40b2-8f10-3fddaf2aebab-kube-api-access-zk6qd\") pod \"payload-processing-e2e-inf-a-286d36-54c688684b-z64mc\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:56.789255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.789227 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275b80bd-da5b-40b2-8f10-3fddaf2aebab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-286d36-54c688684b-z64mc\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:56.812059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.812027 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6qd\" (UniqueName: \"kubernetes.io/projected/275b80bd-da5b-40b2-8f10-3fddaf2aebab-kube-api-access-zk6qd\") pod \"payload-processing-e2e-inf-a-286d36-54c688684b-z64mc\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:56.928802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.928768 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:05:56.932635 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:56.932608 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:57.246621 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:57.246589 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275b80bd_da5b_40b2_8f10_3fddaf2aebab.slice/crio-4b840348007774137afccc69d84a9e9ccb28a94fd099e7936821b0a8a603ae5b WatchSource:0}: Error finding container 4b840348007774137afccc69d84a9e9ccb28a94fd099e7936821b0a8a603ae5b: Status 404 returned error can't find the container with id 4b840348007774137afccc69d84a9e9ccb28a94fd099e7936821b0a8a603ae5b Oct 30 01:05:57.250196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.250172 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc"] Oct 30 01:05:57.738209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.738166 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r"] Oct 30 01:05:57.746078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.746052 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.763554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.763484 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-42d56b-openshift-default-dockercfg-7msn4\"" Oct 30 01:05:57.784692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.784659 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r"] Oct 30 01:05:57.915107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.915060 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-envoy\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.915365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.915342 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-certs\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.915477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.915464 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istiod-ca-cert\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.919758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.919723 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-token\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.919975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.919954 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-credential-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.940755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.940716 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-data\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.941035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.941017 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.941257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.941239 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2sn9\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-kube-api-access-z2sn9\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:57.941532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.941495 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-podinfo\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.001610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:57.997734 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:05:58.043800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.043759 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-podinfo\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.044880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.044856 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-envoy\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.045048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.045031 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-certs\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.045226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.045207 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istiod-ca-cert\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.045532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.045497 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-token\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.045765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.045746 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-credential-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.046914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.045862 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-certs\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.046914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.045916 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-data\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.046914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.046011 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.046914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.046105 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-credential-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.046914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.046147 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2sn9\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-kube-api-access-z2sn9\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.046914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.046334 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.046914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.046572 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-data\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.048223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.048195 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istiod-ca-cert\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.050646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.050274 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-podinfo\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.089491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.089420 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2sn9\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-kube-api-access-z2sn9\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.103850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.103820 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-token\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.126658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.126626 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-envoy\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.130374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.129627 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" event={"ID":"275b80bd-da5b-40b2-8f10-3fddaf2aebab","Type":"ContainerStarted","Data":"6517d993a1009a225d0aa953463683ed47846d8197756ae0f0be73233dfba809"} Oct 30 01:05:58.130374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.129699 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" event={"ID":"275b80bd-da5b-40b2-8f10-3fddaf2aebab","Type":"ContainerStarted","Data":"4b840348007774137afccc69d84a9e9ccb28a94fd099e7936821b0a8a603ae5b"} Oct 30 01:05:58.130374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.130303 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:05:58.245228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.239925 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" podStartSLOduration=2.239902008 podStartE2EDuration="2.239902008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:58.194005451 +0000 UTC m=+1643.137178647" watchObservedRunningTime="2026-10-30 01:05:58.239902008 +0000 UTC m=+1643.183075226" Oct 30 01:05:58.371039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.370955 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:58.724219 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:05:58.724188 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cde45c_ce1c_4b18_81c7_a0edb5afd13a.slice/crio-f1e2860122e32802c101f3ef9e93c78ea51e2b3dbd8180685a8e7a04b065c19f WatchSource:0}: Error finding container f1e2860122e32802c101f3ef9e93c78ea51e2b3dbd8180685a8e7a04b065c19f: Status 404 returned error can't find the container with id f1e2860122e32802c101f3ef9e93c78ea51e2b3dbd8180685a8e7a04b065c19f Oct 30 01:05:58.725713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.725687 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r"] Oct 30 01:05:58.727835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.727798 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:58.727931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.727882 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:58.727994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:58.727926 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:05:59.158307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:59.158269 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" event={"ID":"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a","Type":"ContainerStarted","Data":"1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd"} Oct 30 01:05:59.158777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:59.158359 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" event={"ID":"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a","Type":"ContainerStarted","Data":"f1e2860122e32802c101f3ef9e93c78ea51e2b3dbd8180685a8e7a04b065c19f"} Oct 30 01:05:59.205141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:59.205088 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podStartSLOduration=2.205072273 podStartE2EDuration="2.205072273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:59.203815836 +0000 UTC m=+1644.146989029" watchObservedRunningTime="2026-10-30 01:05:59.205072273 +0000 UTC m=+1644.148245468" Oct 30 01:05:59.373973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:59.373890 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:05:59.432960 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:05:59.432876 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:06:00.169171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:00.169138 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:06:00.170513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:00.170431 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:06:00.983561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:00.983498 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:06:01.993853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:01.993816 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:06:09.162613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:09.162573 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:06:12.416976 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:06:12.416931 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:06:17.304728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:17.303654 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:17.436542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:17.428363 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:17.436542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:17.429840 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:06:17.541617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:17.532088 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:06:17.623202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:17.623091 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n"] Oct 30 01:06:17.631850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:17.631806 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" containerID="cri-o://fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72" gracePeriod=30 Oct 30 01:06:18.419646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:18.419597 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:18.447541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:18.421156 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:06:18.478542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:18.433614 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:18.492660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:18.479317 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:06:18.598282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:18.592805 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:18.817544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:18.807921 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:19.135382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.135273 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:19.139319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.139294 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:06:19.500539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.498675 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:19.547104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.546860 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:19.594687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.594638 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:19.595550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.595504 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:06:19.650312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.650006 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:06:19.650593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:19.650415 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:06:20.161549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.149648 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:20.161549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.150214 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:06:20.485473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.469166 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:20.516547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.504619 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:20.553108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.553063 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:20.553909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.553885 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:06:20.661662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.655656 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:20.661662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.656059 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:20.661662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.656628 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:06:20.958615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.958330 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:20.972548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:20.963067 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:21.158594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.158465 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:06:21.479708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.479660 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:21.486886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.486846 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:06:21.542558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.487140 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 30 01:06:21.543333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.509605 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:06:21.556108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.555617 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:21.691580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.687984 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:21.691580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.689019 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:21.786548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.785199 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:06:21.786548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.785361 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:06:21.797003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.796972 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:06:21.911198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.911166 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:06:21.913785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.913762 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:06:21.950748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.950709 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:21.962229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:21.962178 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:06:22.972275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:22.971872 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 01:06:23.008151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.007596 4905 generic.go:356] "Generic (PLEG): container finished" podID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerID="fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72" exitCode=0 Oct 30 01:06:23.008151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.007732 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" Oct 30 01:06:23.008151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.007943 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" event={"ID":"993d149f-94af-4e5c-8c20-cf4f2f1955e6","Type":"ContainerDied","Data":"fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72"} Oct 30 01:06:23.008151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.007993 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n" event={"ID":"993d149f-94af-4e5c-8c20-cf4f2f1955e6","Type":"ContainerDied","Data":"e0f8bdbfd59b8003973e88a00a46a149fc6930b0fdf8019734778524c8259351"} Oct 30 01:06:23.008151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.008018 4905 scope.go:122] "RemoveContainer" containerID="fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72" Oct 30 01:06:23.024331 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.024302 4905 scope.go:122] "RemoveContainer" containerID="fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72" Oct 30 01:06:23.024777 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:06:23.024742 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72\": container with ID starting with fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72 not found: ID does not exist" containerID="fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72" Oct 30 01:06:23.024979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.024785 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72"} err="failed to get container status \"fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72\": rpc error: code = NotFound desc = could not find container \"fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72\": container with ID starting with fe636722e408ecf5cb3237187d737440db7992e5224b6f2badc222b2b01f8c72 not found: ID does not exist" Oct 30 01:06:23.159711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.154158 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-token\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-token\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.159711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.154237 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-podinfo\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.161276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.161244 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-kube-api-access-h7hqg\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-kube-api-access-h7hqg\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.161459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.161438 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-credential-socket\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.161619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.161535 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istiod-ca-cert\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.161688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.161655 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-data\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.161744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.161691 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-socket\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.161848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.161830 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-envoy\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.161988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.161971 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-certs\") pod \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\" (UID: \"993d149f-94af-4e5c-8c20-cf4f2f1955e6\") " Oct 30 01:06:23.164377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.163354 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-data" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:23.164377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.163378 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istiod-ca-cert" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:23.164377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.163572 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-credential-socket" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:23.164377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.163800 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-socket" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:23.164377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.164019 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-token" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:23.164377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.164051 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-certs" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:23.171308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.171243 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-kube-api-access-h7hqg" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "kube-api-access-h7hqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:23.171823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.171793 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-podinfo" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:23.176698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.176641 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-envoy" pod "993d149f-94af-4e5c-8c20-cf4f2f1955e6" (UID: "993d149f-94af-4e5c-8c20-cf4f2f1955e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:23.273774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273732 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.273774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273768 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.273774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273782 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.274037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273794 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.274037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273807 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7hqg\" (UniqueName: \"kubernetes.io/projected/993d149f-94af-4e5c-8c20-cf4f2f1955e6-kube-api-access-h7hqg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.274037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273820 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.274037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273830 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.274037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273841 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.274037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.273853 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993d149f-94af-4e5c-8c20-cf4f2f1955e6-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.358766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.358726 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n"] Oct 30 01:06:23.365913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:23.365871 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-g9x8n"] Oct 30 01:06:24.594787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:24.594735 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:06:24.595326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:24.594805 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:25.199255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.199218 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" path="/var/lib/kubelet/pods/993d149f-94af-4e5c-8c20-cf4f2f1955e6/volumes" Oct 30 01:06:25.350307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.350257 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm"] Oct 30 01:06:25.351586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.351560 4905 cpu_manager.go:414] "containerMap: removing container" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" Oct 30 01:06:25.351586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.351585 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" Oct 30 01:06:25.351865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.351851 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="993d149f-94af-4e5c-8c20-cf4f2f1955e6" containerName="istio-proxy" Oct 30 01:06:25.392513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.391187 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.408151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.406168 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm"] Oct 30 01:06:25.408497 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.408461 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0680fa-serving-cert\"" Oct 30 01:06:25.416566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.415730 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-metrics-certs\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.416566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.415866 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rjs7\" (UniqueName: \"kubernetes.io/projected/73d96231-b70d-4e2e-8cd3-5390aee5ab62-kube-api-access-2rjs7\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.416566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.415911 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d96231-b70d-4e2e-8cd3-5390aee5ab62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.416566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.416057 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-maas-api-tls\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.518063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.518009 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-metrics-certs\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.518278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.518107 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rjs7\" (UniqueName: \"kubernetes.io/projected/73d96231-b70d-4e2e-8cd3-5390aee5ab62-kube-api-access-2rjs7\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.518278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.518153 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d96231-b70d-4e2e-8cd3-5390aee5ab62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.518438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.518287 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-maas-api-tls\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.519652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.519624 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d96231-b70d-4e2e-8cd3-5390aee5ab62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.521124 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.521092 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-metrics-certs\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.522366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.522343 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-maas-api-tls\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.548462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.548425 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rjs7\" (UniqueName: \"kubernetes.io/projected/73d96231-b70d-4e2e-8cd3-5390aee5ab62-kube-api-access-2rjs7\") pod \"maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:25.747673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:25.740846 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:26.166109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:26.165873 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm"] Oct 30 01:06:26.167854 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:06:26.167805 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d96231_b70d_4e2e_8cd3_5390aee5ab62.slice/crio-c1badd1a2241bb78a517a685cdb50253d5884dc79d2fa04e9cbd358cef1d95d0 WatchSource:0}: Error finding container c1badd1a2241bb78a517a685cdb50253d5884dc79d2fa04e9cbd358cef1d95d0: Status 404 returned error can't find the container with id c1badd1a2241bb78a517a685cdb50253d5884dc79d2fa04e9cbd358cef1d95d0 Oct 30 01:06:27.127839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:27.126257 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" event={"ID":"73d96231-b70d-4e2e-8cd3-5390aee5ab62","Type":"ContainerStarted","Data":"89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168"} Oct 30 01:06:27.127839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:27.126347 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" event={"ID":"73d96231-b70d-4e2e-8cd3-5390aee5ab62","Type":"ContainerStarted","Data":"c1badd1a2241bb78a517a685cdb50253d5884dc79d2fa04e9cbd358cef1d95d0"} Oct 30 01:06:27.127839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:27.127709 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:27.177240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:27.177168 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" podStartSLOduration=1.579733316 podStartE2EDuration="2.177144801s" totalImagesPullingTime="597.411485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:27.175175762 +0000 UTC m=+1672.118348959" watchObservedRunningTime="2026-10-30 01:06:27.177144801 +0000 UTC m=+1672.120317998" Oct 30 01:06:29.154236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.154187 4905 generic.go:356] "Generic (PLEG): container finished" podID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerID="c5fb03af62818c99ff130317894a7288500bec8feebd059fd9f0c5862a9ce362" exitCode=0 Oct 30 01:06:29.154919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.154269 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" event={"ID":"5d0996ac-76a8-4320-ad1d-7cad3c396659","Type":"ContainerDied","Data":"c5fb03af62818c99ff130317894a7288500bec8feebd059fd9f0c5862a9ce362"} Oct 30 01:06:29.155057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.155028 4905 scope.go:122] "RemoveContainer" containerID="c5fb03af62818c99ff130317894a7288500bec8feebd059fd9f0c5862a9ce362" Oct 30 01:06:29.157380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.157347 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.157471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.157434 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.157510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.157479 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.157845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.157810 4905 generic.go:356] "Generic (PLEG): container finished" podID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerID="5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467" exitCode=0 Oct 30 01:06:29.157958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.157887 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" event={"ID":"c1dadcfa-aabe-4763-ac48-e8cd508e4916","Type":"ContainerDied","Data":"5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467"} Oct 30 01:06:29.158462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.158441 4905 scope.go:122] "RemoveContainer" containerID="5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467" Oct 30 01:06:29.160979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.160945 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.161100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.161043 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.161159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.161099 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.164265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.164229 4905 generic.go:356] "Generic (PLEG): container finished" podID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerID="a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e" exitCode=0 Oct 30 01:06:29.164404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.164264 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" event={"ID":"02bb569b-2dc4-48ec-9d22-6b04e2d0f088","Type":"ContainerDied","Data":"a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e"} Oct 30 01:06:29.164772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.164751 4905 scope.go:122] "RemoveContainer" containerID="a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e" Oct 30 01:06:29.173614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.173578 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.173757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.173652 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:29.173757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:29.173687 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:06:30.213543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.212732 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" event={"ID":"5d0996ac-76a8-4320-ad1d-7cad3c396659","Type":"ContainerStarted","Data":"569dcc0f61dbeab789dfe9a0fac2302641e41f5b2cc99da393a6ecd28ed5e5ef"} Oct 30 01:06:30.222306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.222192 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" event={"ID":"c1dadcfa-aabe-4763-ac48-e8cd508e4916","Type":"ContainerStarted","Data":"5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31"} Oct 30 01:06:30.247548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.238549 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" event={"ID":"02bb569b-2dc4-48ec-9d22-6b04e2d0f088","Type":"ContainerStarted","Data":"026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf"} Oct 30 01:06:30.489609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.483412 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:06:30.489609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.483570 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:06:30.489609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.483616 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:06:30.513542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.511675 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:06:30.513542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.511805 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:06:30.513542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:30.511844 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:06:31.036156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:31.014948 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:06:31.038692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:31.038649 4905 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 01:06:31.038860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:31.038712 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 01:06:31.496703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:31.496643 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:06:31.497270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:31.496753 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:06:31.511622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:31.511587 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:06:31.512037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:31.511925 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:06:32.024373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:32.023929 4905 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 01:06:32.024373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:32.023990 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 01:06:32.491608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:32.483387 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:06:32.491608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:32.483456 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:06:32.509548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:32.502042 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:06:32.509548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:32.502106 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:06:33.027565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.012994 4905 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 01:06:33.027565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.013059 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 01:06:33.037292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.036104 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:06:33.063713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.063662 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:33.107048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.107007 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z"] Oct 30 01:06:33.278637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.267422 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22510f0-rw45z"] Oct 30 01:06:33.506552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.506070 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:06:33.506552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.506151 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:06:33.517032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.516994 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:06:33.517690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:33.517643 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:06:34.013309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:34.013218 4905 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 01:06:34.013309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:34.013269 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 01:06:34.258638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:34.255343 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:06:34.531861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:34.496053 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:06:34.531861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:34.496111 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:06:34.556746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:34.556709 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:06:34.615683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:34.615629 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:06:35.046542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.025638 4905 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 01:06:35.046542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.025732 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 01:06:35.424359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.424273 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999218ab-577e-4c0d-816f-12cf38c59666" path="/var/lib/kubelet/pods/999218ab-577e-4c0d-816f-12cf38c59666/volumes" Oct 30 01:06:35.484890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.484853 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:06:35.485191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.485160 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:06:35.501303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.501265 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:06:35.501617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.501586 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:06:35.791672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:35.761639 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:36.013545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.013046 4905 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 01:06:36.013545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.013150 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 01:06:36.109609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.108898 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:06:36.165185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.164287 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:06:36.484184 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.484144 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:06:36.484493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.484463 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:06:36.499188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.499147 4905 patch_prober.go:29] interesting pod/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:06:36.499511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:36.499461 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 01:06:37.019871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.019628 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:06:37.021671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.021642 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:06:37.488922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.488886 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:06:37.490323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.490289 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:06:37.504643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.504613 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:06:37.505006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.504979 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:06:37.505699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.505676 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:06:37.924791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.924676 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp"] Oct 30 01:06:37.963774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:37.963731 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2"] Oct 30 01:06:42.497474 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:06:42.497431 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:06:42.926290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:42.926170 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" podUID="2a2e8ebe-01db-458b-ad90-1622880f0629" containerName="payload-processing" containerID="cri-o://629832c4f25f396831a7750c1cc986fb3d311ccbd16eb002cd2b53c426deb75e" gracePeriod=25 Oct 30 01:06:42.964988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:42.964921 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" podUID="bd292360-d2fd-4c2b-b6a1-208170d8c1f1" containerName="payload-pre-processing" containerID="cri-o://8d24bc5a3e419f1ff207240e6de1d0d864c1621b39fc544ae59f0842daae5c0c" gracePeriod=25 Oct 30 01:06:43.102886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.102845 4905 scope.go:122] "RemoveContainer" containerID="180449f331f3a878849337c9aa55069dc6e1021f916c1c10db7c691261b48b47" Oct 30 01:06:43.118248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.118229 4905 scope.go:122] "RemoveContainer" containerID="90b49f0040a618ce644d1c806df919b10ab109849ed1e3400578404f0a2312e3" Oct 30 01:06:43.133244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.133224 4905 scope.go:122] "RemoveContainer" containerID="c09ab90338466742383a14735c11421b8c5cdc16c569f1cc6a9da61c05603221" Oct 30 01:06:43.155618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.155584 4905 scope.go:122] "RemoveContainer" containerID="b48967f80fb00b72a6008b732bd854c63c2680a27ba583207420d194a75c777c" Oct 30 01:06:43.179100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.178686 4905 scope.go:122] "RemoveContainer" containerID="9bdd584e1da2586ad691312f37d211e2f6fdbb2e2c7270fd68a0445464ffe5fe" Oct 30 01:06:43.215907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.215877 4905 generic.go:356] "Generic (PLEG): container finished" podID="bd292360-d2fd-4c2b-b6a1-208170d8c1f1" containerID="8d24bc5a3e419f1ff207240e6de1d0d864c1621b39fc544ae59f0842daae5c0c" exitCode=0 Oct 30 01:06:43.216178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.215953 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" event={"ID":"bd292360-d2fd-4c2b-b6a1-208170d8c1f1","Type":"ContainerDied","Data":"8d24bc5a3e419f1ff207240e6de1d0d864c1621b39fc544ae59f0842daae5c0c"} Oct 30 01:06:43.221336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.221305 4905 generic.go:356] "Generic (PLEG): container finished" podID="2a2e8ebe-01db-458b-ad90-1622880f0629" containerID="629832c4f25f396831a7750c1cc986fb3d311ccbd16eb002cd2b53c426deb75e" exitCode=0 Oct 30 01:06:43.221501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.221419 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" event={"ID":"2a2e8ebe-01db-458b-ad90-1622880f0629","Type":"ContainerDied","Data":"629832c4f25f396831a7750c1cc986fb3d311ccbd16eb002cd2b53c426deb75e"} Oct 30 01:06:43.322560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.322511 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:06:43.364702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.364666 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2e8ebe-01db-458b-ad90-1622880f0629-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2e8ebe-01db-458b-ad90-1622880f0629-plugins-config-volume\") pod \"2a2e8ebe-01db-458b-ad90-1622880f0629\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " Oct 30 01:06:43.364911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.364765 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2e8ebe-01db-458b-ad90-1622880f0629-kube-api-access-gf8fr\" (UniqueName: \"kubernetes.io/projected/2a2e8ebe-01db-458b-ad90-1622880f0629-kube-api-access-gf8fr\") pod \"2a2e8ebe-01db-458b-ad90-1622880f0629\" (UID: \"2a2e8ebe-01db-458b-ad90-1622880f0629\") " Oct 30 01:06:43.366352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.366316 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2e8ebe-01db-458b-ad90-1622880f0629-plugins-config-volume" pod "2a2e8ebe-01db-458b-ad90-1622880f0629" (UID: "2a2e8ebe-01db-458b-ad90-1622880f0629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:43.368179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.368139 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2e8ebe-01db-458b-ad90-1622880f0629-kube-api-access-gf8fr" pod "2a2e8ebe-01db-458b-ad90-1622880f0629" (UID: "2a2e8ebe-01db-458b-ad90-1622880f0629"). InnerVolumeSpecName "kube-api-access-gf8fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:43.378817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.378775 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:06:43.577644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.563985 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2e8ebe-01db-458b-ad90-1622880f0629-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:43.577644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.564017 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf8fr\" (UniqueName: \"kubernetes.io/projected/2a2e8ebe-01db-458b-ad90-1622880f0629-kube-api-access-gf8fr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:43.677216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.677169 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-kube-api-access-4q9d7\" (UniqueName: \"kubernetes.io/projected/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-kube-api-access-4q9d7\") pod \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " Oct 30 01:06:43.688817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.688778 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-plugins-config-volume\") pod \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\" (UID: \"bd292360-d2fd-4c2b-b6a1-208170d8c1f1\") " Oct 30 01:06:43.740546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.738506 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-kube-api-access-4q9d7" pod "bd292360-d2fd-4c2b-b6a1-208170d8c1f1" (UID: "bd292360-d2fd-4c2b-b6a1-208170d8c1f1"). InnerVolumeSpecName "kube-api-access-4q9d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:43.745273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.745185 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-plugins-config-volume" pod "bd292360-d2fd-4c2b-b6a1-208170d8c1f1" (UID: "bd292360-d2fd-4c2b-b6a1-208170d8c1f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:43.762210 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.761067 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:43.762210 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.761100 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q9d7\" (UniqueName: \"kubernetes.io/projected/bd292360-d2fd-4c2b-b6a1-208170d8c1f1-kube-api-access-4q9d7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:06:43.819866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.819824 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc"] Oct 30 01:06:43.821163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.821141 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2a2e8ebe-01db-458b-ad90-1622880f0629" containerName="payload-processing" Oct 30 01:06:43.821330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.821319 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2e8ebe-01db-458b-ad90-1622880f0629" containerName="payload-processing" Oct 30 01:06:43.821403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.821396 4905 cpu_manager.go:414] "containerMap: removing container" podUID="bd292360-d2fd-4c2b-b6a1-208170d8c1f1" containerName="payload-pre-processing" Oct 30 01:06:43.821454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.821447 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd292360-d2fd-4c2b-b6a1-208170d8c1f1" containerName="payload-pre-processing" Oct 30 01:06:43.821684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.821674 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd292360-d2fd-4c2b-b6a1-208170d8c1f1" containerName="payload-pre-processing" Oct 30 01:06:43.821775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.821768 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2e8ebe-01db-458b-ad90-1622880f0629" containerName="payload-processing" Oct 30 01:06:43.825047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.825022 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.848232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.845284 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-42d56b-serving-cert\"" Oct 30 01:06:43.852851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.852825 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc"] Oct 30 01:06:43.865114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.865070 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knfqw\" (UniqueName: \"kubernetes.io/projected/dc1d798f-f790-41d3-b9f0-773cf307b368-kube-api-access-knfqw\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.865479 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.865452 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.865770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.865747 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d798f-f790-41d3-b9f0-773cf307b368-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.865980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.865964 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.967471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.967438 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.967706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.967499 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d798f-f790-41d3-b9f0-773cf307b368-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.967706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.967572 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.967909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.967892 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knfqw\" (UniqueName: \"kubernetes.io/projected/dc1d798f-f790-41d3-b9f0-773cf307b368-kube-api-access-knfqw\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.968228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.968199 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d798f-f790-41d3-b9f0-773cf307b368-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.970577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.970238 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.970577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.970431 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:43.988903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:43.988871 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knfqw\" (UniqueName: \"kubernetes.io/projected/dc1d798f-f790-41d3-b9f0-773cf307b368-kube-api-access-knfqw\") pod \"maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:44.175185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.174406 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:44.272545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.271577 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" event={"ID":"bd292360-d2fd-4c2b-b6a1-208170d8c1f1","Type":"ContainerDied","Data":"7a6984b9ac531307f973c7cdf83430f04e6144d2cf172783a012bd7c7cf490bd"} Oct 30 01:06:44.272545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.271680 4905 scope.go:122] "RemoveContainer" containerID="8d24bc5a3e419f1ff207240e6de1d0d864c1621b39fc544ae59f0842daae5c0c" Oct 30 01:06:44.272545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.271939 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" Oct 30 01:06:44.333949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.333924 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" Oct 30 01:06:44.334195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.333952 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp" event={"ID":"2a2e8ebe-01db-458b-ad90-1622880f0629","Type":"ContainerDied","Data":"2083c4bcecbf42883d3e1ff28cf46310febeb81dfdffbce2f6818818a00d69a7"} Oct 30 01:06:44.349097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.349070 4905 scope.go:122] "RemoveContainer" containerID="629832c4f25f396831a7750c1cc986fb3d311ccbd16eb002cd2b53c426deb75e" Oct 30 01:06:44.359359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.358929 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2" podUID="bd292360-d2fd-4c2b-b6a1-208170d8c1f1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.199:50052\" within 1s: context deadline exceeded" Oct 30 01:06:44.359359 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:06:44.359107 4905 logging.go:55] [core] [Channel #2863 SubChannel #2864]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.199:50052", ServerName: "10.132.0.199:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.199:50052: operation was canceled" Oct 30 01:06:44.389731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.389684 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2"] Oct 30 01:06:44.406552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.406496 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c49c0c-6855847799-s2pm2"] Oct 30 01:06:44.435615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.435439 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp"] Oct 30 01:06:44.443946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.443911 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c49c0c-6b8687b998-d5fzp"] Oct 30 01:06:44.657348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:44.657306 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc"] Oct 30 01:06:45.195952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:45.195917 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2e8ebe-01db-458b-ad90-1622880f0629" path="/var/lib/kubelet/pods/2a2e8ebe-01db-458b-ad90-1622880f0629/volumes" Oct 30 01:06:45.196435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:45.196419 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd292360-d2fd-4c2b-b6a1-208170d8c1f1" path="/var/lib/kubelet/pods/bd292360-d2fd-4c2b-b6a1-208170d8c1f1/volumes" Oct 30 01:06:45.350081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:45.350042 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" event={"ID":"dc1d798f-f790-41d3-b9f0-773cf307b368","Type":"ContainerStarted","Data":"301d9d1696ac826256be2feeb65df9b77d71af23d0d24bcf69b487a73211413c"} Oct 30 01:06:45.350308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:45.350104 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" event={"ID":"dc1d798f-f790-41d3-b9f0-773cf307b368","Type":"ContainerStarted","Data":"5d2c5cbe921ad577fb4a43a0d41d64d1181f36f166162c14f992d74079de98b0"} Oct 30 01:06:45.350308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:45.350158 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:06:45.428737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:45.412917 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" podStartSLOduration=1.930282525 podStartE2EDuration="2.412895643s" totalImagesPullingTime="482.613118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:45.410640097 +0000 UTC m=+1690.353813292" watchObservedRunningTime="2026-10-30 01:06:45.412895643 +0000 UTC m=+1690.356068836" Oct 30 01:06:48.224549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.219806 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g"] Oct 30 01:06:48.256540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.256063 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.295539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.290448 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g"] Oct 30 01:06:48.310994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.310962 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1eae0b\"" Oct 30 01:06:48.311188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.311124 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1eae0b-dockercfg-bn8kp\"" Oct 30 01:06:48.411309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.411270 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrxk\" (UniqueName: \"kubernetes.io/projected/4139be30-705e-403f-944b-8013a773f730-kube-api-access-fbrxk\") pod \"payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.411493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.411470 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4139be30-705e-403f-944b-8013a773f730-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.512839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.512800 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4139be30-705e-403f-944b-8013a773f730-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.513057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.513019 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrxk\" (UniqueName: \"kubernetes.io/projected/4139be30-705e-403f-944b-8013a773f730-kube-api-access-fbrxk\") pod \"payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.513733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.513712 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4139be30-705e-403f-944b-8013a773f730-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.535044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.535015 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrxk\" (UniqueName: \"kubernetes.io/projected/4139be30-705e-403f-944b-8013a773f730-kube-api-access-fbrxk\") pod \"payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.640541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.634305 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:48.980945 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:06:48.980911 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4139be30_705e_403f_944b_8013a773f730.slice/crio-d94f2c94d06ee25dbb971a9c78103f705fcdee0bfdc3d20d51b1afc8eaa9f0cf WatchSource:0}: Error finding container d94f2c94d06ee25dbb971a9c78103f705fcdee0bfdc3d20d51b1afc8eaa9f0cf: Status 404 returned error can't find the container with id d94f2c94d06ee25dbb971a9c78103f705fcdee0bfdc3d20d51b1afc8eaa9f0cf Oct 30 01:06:48.983821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:48.983787 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g"] Oct 30 01:06:49.393214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:49.393167 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" event={"ID":"4139be30-705e-403f-944b-8013a773f730","Type":"ContainerStarted","Data":"e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8"} Oct 30 01:06:49.393670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:49.393228 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" event={"ID":"4139be30-705e-403f-944b-8013a773f730","Type":"ContainerStarted","Data":"d94f2c94d06ee25dbb971a9c78103f705fcdee0bfdc3d20d51b1afc8eaa9f0cf"} Oct 30 01:06:49.393670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:49.393249 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:06:49.440633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:49.440552 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" podStartSLOduration=2.440531141 podStartE2EDuration="2.440531141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:49.435800352 +0000 UTC m=+1694.378973547" watchObservedRunningTime="2026-10-30 01:06:49.440531141 +0000 UTC m=+1694.383704326" Oct 30 01:06:51.370319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:06:51.370275 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:07:00.493547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:00.492333 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:07:09.053217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.053118 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:07:09.054278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.054152 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:09.127559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.121784 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9"] Oct 30 01:07:09.178609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.154256 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7be503-rjsk9"] Oct 30 01:07:09.265553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.264323 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce" path="/var/lib/kubelet/pods/8f5f4a08-94c3-49dc-9cf4-8dc0b29e88ce/volumes" Oct 30 01:07:09.267977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.267939 4905 generic.go:356] "Generic (PLEG): container finished" podID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerID="9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8" exitCode=0 Oct 30 01:07:09.268200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.268180 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" event={"ID":"74f3e0d6-a86c-4527-ab11-e6732c123c73","Type":"ContainerDied","Data":"9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8"} Oct 30 01:07:09.268816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.268794 4905 scope.go:122] "RemoveContainer" containerID="9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8" Oct 30 01:07:09.271645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.271603 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:09.271839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.271725 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:09.271839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:09.271781 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:10.279414 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.279378 4905 generic.go:356] "Generic (PLEG): container finished" podID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerID="3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068" exitCode=0 Oct 30 01:07:10.279903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.279575 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" event={"ID":"7ef45997-65ed-4532-8fb2-ce882d7ec527","Type":"ContainerDied","Data":"3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068"} Oct 30 01:07:10.280101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.280083 4905 scope.go:122] "RemoveContainer" containerID="3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068" Oct 30 01:07:10.282172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.282137 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:10.282258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.282215 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:10.282258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.282250 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:10.285423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.285130 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" event={"ID":"74f3e0d6-a86c-4527-ab11-e6732c123c73","Type":"ContainerStarted","Data":"11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564"} Oct 30 01:07:10.286796 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.286772 4905 generic.go:356] "Generic (PLEG): container finished" podID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerID="e4d04365958fa41c2c086106555bc822b2e86567a3850358a525a1dd469cc708" exitCode=0 Oct 30 01:07:10.287090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.286841 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" event={"ID":"54ff57b2-7fb3-40bb-a720-7c35964885f2","Type":"ContainerDied","Data":"e4d04365958fa41c2c086106555bc822b2e86567a3850358a525a1dd469cc708"} Oct 30 01:07:10.287210 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.287197 4905 scope.go:122] "RemoveContainer" containerID="e4d04365958fa41c2c086106555bc822b2e86567a3850358a525a1dd469cc708" Oct 30 01:07:10.295144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.295104 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:10.295279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.295199 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:10.295279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.295244 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:10.394625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.394593 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:07:10.395761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.395738 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:10.395971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.395947 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:10.706598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:10.701659 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:07:11.304250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.304204 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" event={"ID":"7ef45997-65ed-4532-8fb2-ce882d7ec527","Type":"ContainerStarted","Data":"1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879"} Oct 30 01:07:11.315748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.315709 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" event={"ID":"54ff57b2-7fb3-40bb-a720-7c35964885f2","Type":"ContainerStarted","Data":"082f7eda2fe1948bded6d130ac1da1569927b677648d5af28418cfaac57abe8c"} Oct 30 01:07:11.396561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.395682 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:11.396561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.395756 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:11.446546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.437478 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:07:11.446546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.438634 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:11.446546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.438710 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:11.801619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.793893 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:07:11.801619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.799692 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:11.801619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:11.799749 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:12.409548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.406265 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:12.409548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.406373 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:12.458721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.450452 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:12.458721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.450545 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:12.520013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.519943 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:12.595713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.579631 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:12.681615 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:07:12.664325 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:07:12.730554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.728733 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:07:12.799577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.799538 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:12.800511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.800479 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:12.801091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.799800 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:07:12.801499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.799747 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:12.855712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:12.855664 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:13.232551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.223752 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:13.401546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.400495 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:13.401546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.400581 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:13.452548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.431212 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:13.452548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.431273 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:13.454145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.454103 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:13.603612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.602942 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:13.603837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.603735 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:07:13.840580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.824873 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:07:13.872130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.861636 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:07:13.872130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.862114 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:13.872130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.862167 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:13.872130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:13.870985 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:07:14.509549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.458025 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:14.509549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.458083 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:14.576627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.540825 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:14.576627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.540899 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:14.634679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.634629 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:14.799199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.799108 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:14.799450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.799423 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:14.823502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.822602 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:07:14.851581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.845738 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:14.851581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.846079 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:14.851581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.846258 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:07:14.872378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.862614 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:14.872378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:14.871685 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:07:15.122832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.122710 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:15.151798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.149629 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:15.396016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.395916 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:15.396302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.396270 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:15.429967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.429926 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:15.430299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.430266 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:15.517243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.517197 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:07:15.616460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.616411 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:15.640727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.640671 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:15.641805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.641775 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:07:15.734609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.696936 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:07:15.756548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.735834 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:15.787626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.787586 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:15.809090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.809040 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:15.850870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.850821 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:07:15.882675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.862809 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:15.882903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.882760 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:07:15.883000 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.882991 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:07:15.912563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.896588 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:07:15.912563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:15.896992 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:07:16.138963 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.137656 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:16.138963 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.138278 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:07:16.400612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.328209 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:16.401259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.395986 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:16.493549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.492557 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:16.518800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.518765 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:07:16.659613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.644137 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:07:16.792545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.770534 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 01:07:16.801262 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.798636 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:16.923651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.801656 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:16.958546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.806737 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:16.958546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.931067 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:16.958546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.866239 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:16.958546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.827159 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:16.958546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.939182 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:17.001260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.974280 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:17.001260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.974370 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:17.001260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:16.987918 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:07:17.002076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.002034 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:17.070547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.059282 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:07:17.070547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.059377 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:07:17.158008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.157932 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:17.452706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.452651 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:17.686903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.686859 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:17.687539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.687486 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:17.717604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.711931 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 01:07:17.762550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.753639 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:07:17.762550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.753711 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 01:07:17.793546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.792849 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:17.802548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.797779 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:17.802548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.797835 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:17.823003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.822967 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:07:17.823277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.823263 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t"] Oct 30 01:07:17.923706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.923655 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 01:07:18.004546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:17.990376 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:07:18.029344 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.009093 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.031790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.031734 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:07:18.178042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.177983 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.262129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.262044 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:07:18.353037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.352974 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:18.355854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.355828 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t"] Oct 30 01:07:18.428549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.400319 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b2bf8a\"" Oct 30 01:07:18.428549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.400465 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b2bf8a-dockercfg-d4jf2\"" Oct 30 01:07:18.439017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.438979 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:07:18.439306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.439273 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:07:18.623608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.618955 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:18.633547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.632245 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2k5\" (UniqueName: \"kubernetes.io/projected/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-kube-api-access-8p2k5\") pod \"payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:18.686586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.684475 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:07:18.705648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.705620 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:07:18.715665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.715609 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.744555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.742479 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:18.744555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.742606 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2k5\" (UniqueName: \"kubernetes.io/projected/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-kube-api-access-8p2k5\") pod \"payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:18.744555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.743836 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:18.797094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.797054 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 01:07:18.797486 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.797457 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 01:07:18.798142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.797374 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.847820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.847770 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.866369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.866332 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2k5\" (UniqueName: \"kubernetes.io/projected/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-kube-api-access-8p2k5\") pod \"payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:18.893729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.888234 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:07:18.919410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.904487 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:07:18.920185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.920144 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:07:18.926082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.925856 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.976836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:18.976796 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:19.009647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.007981 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:07:19.023064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.023030 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:07:19.033167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.033126 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:07:19.116930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.116895 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:07:19.179193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.178796 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:19.248722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.248681 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:07:19.264589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.264558 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:07:19.332390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.332358 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:07:19.372651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.338885 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:07:19.372651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.339891 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:07:19.594900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.591062 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:07:19.594900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.594365 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:07:19.613755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.610263 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t"] Oct 30 01:07:19.613755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.611239 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:07:19.785679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.785645 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:07:19.786196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.786178 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:07:19.786888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:19.786866 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:07:20.153254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:20.153215 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" event={"ID":"83dd55cc-49d0-4b6b-a896-95a6e3573ef6","Type":"ContainerStarted","Data":"910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7"} Oct 30 01:07:20.153470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:20.153275 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" event={"ID":"83dd55cc-49d0-4b6b-a896-95a6e3573ef6","Type":"ContainerStarted","Data":"21f12d1053243ff2c92b447527d09b752326272eb1deb315a5a6b4e1920d39b6"} Oct 30 01:07:20.153470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:20.153299 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:20.196279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:20.196220 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" podStartSLOduration=3.196203198 podStartE2EDuration="3.196203198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:20.193723508 +0000 UTC m=+1725.136896701" watchObservedRunningTime="2026-10-30 01:07:20.196203198 +0000 UTC m=+1725.139376432" Oct 30 01:07:21.895377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:21.895343 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb"] Oct 30 01:07:21.899357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:21.899322 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:21.917423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:21.917397 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb"] Oct 30 01:07:21.953452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:21.953421 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d54608\"" Oct 30 01:07:21.960324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:21.959551 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqzb\" (UniqueName: \"kubernetes.io/projected/21efb889-95cc-411b-9da7-071f0203eb8a-kube-api-access-sbqzb\") pod \"payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:21.960324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:21.960039 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21efb889-95cc-411b-9da7-071f0203eb8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:22.067546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:22.063099 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21efb889-95cc-411b-9da7-071f0203eb8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:22.067546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:22.063217 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqzb\" (UniqueName: \"kubernetes.io/projected/21efb889-95cc-411b-9da7-071f0203eb8a-kube-api-access-sbqzb\") pod \"payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:22.067546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:22.064693 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21efb889-95cc-411b-9da7-071f0203eb8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:22.404007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:22.403969 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqzb\" (UniqueName: \"kubernetes.io/projected/21efb889-95cc-411b-9da7-071f0203eb8a-kube-api-access-sbqzb\") pod \"payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:22.517133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:22.517103 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:22.821437 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:07:22.821403 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21efb889_95cc_411b_9da7_071f0203eb8a.slice/crio-3657af1f5d4be7675daf363ee577c0962f7791490232d0604f6cc1a649a31bec WatchSource:0}: Error finding container 3657af1f5d4be7675daf363ee577c0962f7791490232d0604f6cc1a649a31bec: Status 404 returned error can't find the container with id 3657af1f5d4be7675daf363ee577c0962f7791490232d0604f6cc1a649a31bec Oct 30 01:07:22.823316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:22.823292 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb"] Oct 30 01:07:23.379179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:23.379142 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" event={"ID":"21efb889-95cc-411b-9da7-071f0203eb8a","Type":"ContainerStarted","Data":"bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e"} Oct 30 01:07:23.379631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:23.379198 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" event={"ID":"21efb889-95cc-411b-9da7-071f0203eb8a","Type":"ContainerStarted","Data":"3657af1f5d4be7675daf363ee577c0962f7791490232d0604f6cc1a649a31bec"} Oct 30 01:07:23.379631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:23.379252 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:23.423381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:23.423320 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" podStartSLOduration=2.423305162 podStartE2EDuration="2.423305162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:23.41923553 +0000 UTC m=+1728.362408724" watchObservedRunningTime="2026-10-30 01:07:23.423305162 +0000 UTC m=+1728.366478355" Oct 30 01:07:27.261442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.261396 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww"] Oct 30 01:07:27.268333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.268303 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.288621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.288587 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww"] Oct 30 01:07:27.294243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.294215 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bc6092-openshift-default-dockercfg-zrj92\"" Oct 30 01:07:27.329033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.328993 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxjn\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-kube-api-access-smxjn\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329047 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-envoy\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329099 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-data\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329186 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f280e47e-36f4-4451-a458-311662858ecd-istio-podinfo\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329230 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f280e47e-36f4-4451-a458-311662858ecd-istiod-ca-cert\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329249 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329373 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-certs\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329446 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-istio-token\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.329584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.329496 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-credential-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.455870 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxjn\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-kube-api-access-smxjn\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.455933 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-envoy\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.455986 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-data\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.456097 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f280e47e-36f4-4451-a458-311662858ecd-istio-podinfo\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.456145 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f280e47e-36f4-4451-a458-311662858ecd-istiod-ca-cert\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.456173 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.456385 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-certs\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.456587 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-istio-token\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.456668 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-credential-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.458544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.457222 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-credential-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.517856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.469402 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f280e47e-36f4-4451-a458-311662858ecd-istiod-ca-cert\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.517856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.469703 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-data\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.517856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.489573 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.517856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.517704 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-certs\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.520551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.519210 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f280e47e-36f4-4451-a458-311662858ecd-istio-podinfo\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.520551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.519862 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-envoy\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.545547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.534135 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-istio-token\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.546342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.546264 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxjn\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-kube-api-access-smxjn\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:27.640546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:27.619907 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:28.109993 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:07:28.109940 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf280e47e_36f4_4451_a458_311662858ecd.slice/crio-a4ce67d265a55ceb2be84fe9b76f83ad842c2ccd4247567051b3a548f21c3f77 WatchSource:0}: Error finding container a4ce67d265a55ceb2be84fe9b76f83ad842c2ccd4247567051b3a548f21c3f77: Status 404 returned error can't find the container with id a4ce67d265a55ceb2be84fe9b76f83ad842c2ccd4247567051b3a548f21c3f77 Oct 30 01:07:28.110882 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.110855 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww"] Oct 30 01:07:28.113650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.113612 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:28.113763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.113706 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:28.113832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.113782 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:28.603003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.602965 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" event={"ID":"f280e47e-36f4-4451-a458-311662858ecd","Type":"ContainerStarted","Data":"13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd"} Oct 30 01:07:28.603503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.603040 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" event={"ID":"f280e47e-36f4-4451-a458-311662858ecd","Type":"ContainerStarted","Data":"a4ce67d265a55ceb2be84fe9b76f83ad842c2ccd4247567051b3a548f21c3f77"} Oct 30 01:07:28.622065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.622036 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:28.677484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:28.677425 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podStartSLOduration=1.677408558 podStartE2EDuration="1.677408558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:28.661480676 +0000 UTC m=+1733.604653869" watchObservedRunningTime="2026-10-30 01:07:28.677408558 +0000 UTC m=+1733.620581752" Oct 30 01:07:29.628974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:29.628940 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:30.622390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:30.622343 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:30.623315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:30.623290 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:07:31.165462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:31.165431 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:07:34.357357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.357317 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4"] Oct 30 01:07:34.364487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.364455 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.383468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.383413 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4"] Oct 30 01:07:34.390243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.390216 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-d0ae0d-openshift-default-dockercfg-2cc5d\"" Oct 30 01:07:34.408435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.408388 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:07:34.462311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462269 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.462311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462325 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.462676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462491 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.462676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462588 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70491b5d-60ce-4630-997e-9f16b05bc8b4-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.462676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462624 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqstz\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-kube-api-access-nqstz\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.462676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462673 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.462962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462742 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.462962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.462844 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.463130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.463109 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.565687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.565642 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.565903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.565733 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70491b5d-60ce-4630-997e-9f16b05bc8b4-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.565903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.565766 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqstz\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-kube-api-access-nqstz\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.565903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.565809 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.565903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.565855 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.566165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.565946 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.566165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.566146 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.566283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.566214 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.566283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.566241 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.572264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.567048 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.572264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.567665 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70491b5d-60ce-4630-997e-9f16b05bc8b4-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.572264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.568115 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.572264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.568327 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.572264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.569134 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.572264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.571712 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.573863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.573817 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.587567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.587512 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.591416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.591388 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqstz\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-kube-api-access-nqstz\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:34.686784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:34.686686 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:35.263600 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:07:35.263558 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70491b5d_60ce_4630_997e_9f16b05bc8b4.slice/crio-e62109b6578f5b7c2abb2f78425ee32014fea985829afb278d41526153d16fed WatchSource:0}: Error finding container e62109b6578f5b7c2abb2f78425ee32014fea985829afb278d41526153d16fed: Status 404 returned error can't find the container with id e62109b6578f5b7c2abb2f78425ee32014fea985829afb278d41526153d16fed Oct 30 01:07:35.269079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:35.269035 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:35.274510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:35.274476 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:35.274762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:35.274729 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:35.277116 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:35.277095 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4"] Oct 30 01:07:35.749983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:35.749939 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" event={"ID":"70491b5d-60ce-4630-997e-9f16b05bc8b4","Type":"ContainerStarted","Data":"f474304491c598d8f17f6176621622605723fd495ae5aceb845b69ea8be1f0f1"} Oct 30 01:07:35.750553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:35.750013 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" event={"ID":"70491b5d-60ce-4630-997e-9f16b05bc8b4","Type":"ContainerStarted","Data":"e62109b6578f5b7c2abb2f78425ee32014fea985829afb278d41526153d16fed"} Oct 30 01:07:35.796089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:35.796036 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podStartSLOduration=1.796022343 podStartE2EDuration="1.796022343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:35.793328 +0000 UTC m=+1740.736501193" watchObservedRunningTime="2026-10-30 01:07:35.796022343 +0000 UTC m=+1740.739195536" Oct 30 01:07:36.687721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:36.687683 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:36.695969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:36.695938 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:36.762433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:36.762390 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:36.764087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:36.764029 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:07:40.259676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.259641 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk"] Oct 30 01:07:40.266320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.266290 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:40.287754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.287708 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk"] Oct 30 01:07:40.289289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.289252 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-d0ae0d-dockercfg-rgcgw\"" Oct 30 01:07:40.305947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.305913 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-d0ae0d\"" Oct 30 01:07:40.388598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.387021 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknpz\" (UniqueName: \"kubernetes.io/projected/4931e93a-0f20-47c1-9f73-ffe763e183b1-kube-api-access-nknpz\") pod \"payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:40.399553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.395834 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4931e93a-0f20-47c1-9f73-ffe763e183b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:40.718643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.716404 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknpz\" (UniqueName: \"kubernetes.io/projected/4931e93a-0f20-47c1-9f73-ffe763e183b1-kube-api-access-nknpz\") pod \"payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:40.718643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.716573 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4931e93a-0f20-47c1-9f73-ffe763e183b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:40.718643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.717813 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4931e93a-0f20-47c1-9f73-ffe763e183b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:40.766793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.766753 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknpz\" (UniqueName: \"kubernetes.io/projected/4931e93a-0f20-47c1-9f73-ffe763e183b1-kube-api-access-nknpz\") pod \"payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:40.883170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:40.883127 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:41.256808 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:07:41.256773 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4931e93a_0f20_47c1_9f73_ffe763e183b1.slice/crio-500d157b0792090b35c8125a4288a88dfd717cd50cfa70249165106432d071c6 WatchSource:0}: Error finding container 500d157b0792090b35c8125a4288a88dfd717cd50cfa70249165106432d071c6: Status 404 returned error can't find the container with id 500d157b0792090b35c8125a4288a88dfd717cd50cfa70249165106432d071c6 Oct 30 01:07:41.260128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:41.260097 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk"] Oct 30 01:07:41.821562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:41.821480 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" event={"ID":"4931e93a-0f20-47c1-9f73-ffe763e183b1","Type":"ContainerStarted","Data":"e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458"} Oct 30 01:07:41.821562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:41.821565 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" event={"ID":"4931e93a-0f20-47c1-9f73-ffe763e183b1","Type":"ContainerStarted","Data":"500d157b0792090b35c8125a4288a88dfd717cd50cfa70249165106432d071c6"} Oct 30 01:07:41.821851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:41.821595 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:41.870259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:41.870200 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" podStartSLOduration=1.870183071 podStartE2EDuration="1.870183071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:41.868762872 +0000 UTC m=+1746.811936068" watchObservedRunningTime="2026-10-30 01:07:41.870183071 +0000 UTC m=+1746.813356265" Oct 30 01:07:42.420752 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:07:42.420719 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:07:43.301983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:43.301954 4905 scope.go:122] "RemoveContainer" containerID="59f6c5301b45921783e583036ceb95f3d141ce52ab87eca629b6af48c593979b" Oct 30 01:07:46.211785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:46.211676 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:07:46.211785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:46.211766 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:48.095895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:48.095858 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq"] Oct 30 01:07:48.183809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:48.183769 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1d19f7f-jrhwq"] Oct 30 01:07:49.194404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:49.194357 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7561b2a-41d8-4238-8627-0eb82e85794c" path="/var/lib/kubelet/pods/d7561b2a-41d8-4238-8627-0eb82e85794c/volumes" Oct 30 01:07:49.974256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:49.974172 4905 generic.go:356] "Generic (PLEG): container finished" podID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerID="1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd" exitCode=0 Oct 30 01:07:49.974477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:49.974267 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" event={"ID":"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a","Type":"ContainerDied","Data":"1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd"} Oct 30 01:07:49.974890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:49.974864 4905 scope.go:122] "RemoveContainer" containerID="1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd" Oct 30 01:07:49.977603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:49.977573 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:49.977703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:49.977648 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:49.977703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:49.977691 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:07:50.985212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:50.985179 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" event={"ID":"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a","Type":"ContainerStarted","Data":"7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb"} Oct 30 01:07:51.371565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:51.371461 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:07:51.371764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:51.371657 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 01:07:51.371764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:51.371722 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 01:07:52.371573 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:52.371537 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 01:07:52.371991 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:52.371599 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 01:07:52.873724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:52.873693 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:07:53.372126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:53.372086 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 01:07:53.372763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:53.372717 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 01:07:54.371340 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:54.371294 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 01:07:54.371833 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:54.371373 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 01:07:55.298545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:55.290891 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth"] Oct 30 01:07:55.298545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:55.291135 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" containerID="cri-o://44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62" gracePeriod=30 Oct 30 01:07:55.375547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:55.375273 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 01:07:55.375547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:55.375344 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 01:07:56.371281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:56.371246 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 01:07:56.371747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:56.371309 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 01:07:57.376619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:57.376589 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:07:57.377204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:57.377179 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:07:57.378055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:07:57.378033 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:08:00.626362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.625850 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:08:00.749095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749046 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-credential-socket\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749135 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-token\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-token\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749168 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istiod-ca-cert\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749294 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-certs\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749365 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-credential-socket" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:00.749445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749377 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-kube-api-access-5jbp4\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-kube-api-access-5jbp4\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749465 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-podinfo\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749674 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-envoy\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749756 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-data\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.749970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749951 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-socket\") pod \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\" (UID: \"fffa3510-3a48-4aff-b299-584ce5e7a0ea\") " Oct 30 01:08:00.750164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.749983 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-data" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:00.750164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.750151 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-socket" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:00.750298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.750005 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-certs" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:00.750472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.750450 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istiod-ca-cert" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:00.751392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.751368 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.751582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.751566 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.751864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.751847 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.751977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.751962 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.752078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.752065 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.752360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.752333 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-kube-api-access-5jbp4" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "kube-api-access-5jbp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:00.754185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.754159 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-envoy" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:00.755959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.755932 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-podinfo" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:00.756084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.756063 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-token" pod "fffa3510-3a48-4aff-b299-584ce5e7a0ea" (UID: "fffa3510-3a48-4aff-b299-584ce5e7a0ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:00.853862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.853775 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbp4\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-kube-api-access-5jbp4\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.853862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.853808 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.853862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.853821 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:00.853862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:00.853833 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffa3510-3a48-4aff-b299-584ce5e7a0ea-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:01.395567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.394643 4905 generic.go:356] "Generic (PLEG): container finished" podID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerID="44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62" exitCode=0 Oct 30 01:08:01.395567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.394724 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" Oct 30 01:08:01.395567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.394702 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" event={"ID":"fffa3510-3a48-4aff-b299-584ce5e7a0ea","Type":"ContainerDied","Data":"44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62"} Oct 30 01:08:01.395567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.395053 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth" event={"ID":"fffa3510-3a48-4aff-b299-584ce5e7a0ea","Type":"ContainerDied","Data":"58320035bb7245ce123401831198fbe578f71b951fa3a5142bd59dc5aaf7eb13"} Oct 30 01:08:01.395567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.395083 4905 scope.go:122] "RemoveContainer" containerID="44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62" Oct 30 01:08:01.415624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.414408 4905 scope.go:122] "RemoveContainer" containerID="db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42" Oct 30 01:08:01.433014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.432990 4905 scope.go:122] "RemoveContainer" containerID="44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62" Oct 30 01:08:01.434320 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:08:01.434288 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62\": container with ID starting with 44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62 not found: ID does not exist" containerID="44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62" Oct 30 01:08:01.434416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.434339 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62"} err="failed to get container status \"44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62\": rpc error: code = NotFound desc = could not find container \"44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62\": container with ID starting with 44f8e068946784923504e6b15dfabddee36bc09d135dae2d7ffd8ad8ac9e2a62 not found: ID does not exist" Oct 30 01:08:01.434416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.434359 4905 scope.go:122] "RemoveContainer" containerID="db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42" Oct 30 01:08:01.435045 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:08:01.435016 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42\": container with ID starting with db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42 not found: ID does not exist" containerID="db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42" Oct 30 01:08:01.435131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.435060 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42"} err="failed to get container status \"db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42\": rpc error: code = NotFound desc = could not find container \"db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42\": container with ID starting with db8e1b7021c1f199f43dfee00d0ce57b60273695db0de62ae1840ce357098c42 not found: ID does not exist" Oct 30 01:08:01.467314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.467283 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth"] Oct 30 01:08:01.480819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:01.480791 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c49c0c-openshift-default-86bbd545d9-l2pth"] Oct 30 01:08:03.194782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:03.194748 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" path="/var/lib/kubelet/pods/fffa3510-3a48-4aff-b299-584ce5e7a0ea/volumes" Oct 30 01:08:05.997987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:05.997952 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx"] Oct 30 01:08:06.033152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:06.033113 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q"] Oct 30 01:08:10.998813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:10.998761 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" podUID="f5a9ee6a-f443-44b4-b8a9-32ed357beae9" containerName="payload-processing" containerID="cri-o://fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963" gracePeriod=25 Oct 30 01:08:11.033703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.033654 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" podUID="c5691598-7eb7-449d-a4cf-b905f1db9cd5" containerName="payload-pre-processing" containerID="cri-o://5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8" gracePeriod=25 Oct 30 01:08:11.358481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.358391 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:08:11.392126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.392090 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-plugins-config-volume\") pod \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " Oct 30 01:08:11.392420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.392392 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-kube-api-access-s8tbk\" (UniqueName: \"kubernetes.io/projected/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-kube-api-access-s8tbk\") pod \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\" (UID: \"f5a9ee6a-f443-44b4-b8a9-32ed357beae9\") " Oct 30 01:08:11.392619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.392497 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-plugins-config-volume" pod "f5a9ee6a-f443-44b4-b8a9-32ed357beae9" (UID: "f5a9ee6a-f443-44b4-b8a9-32ed357beae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:11.392994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.392975 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:11.395053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.394799 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-kube-api-access-s8tbk" pod "f5a9ee6a-f443-44b4-b8a9-32ed357beae9" (UID: "f5a9ee6a-f443-44b4-b8a9-32ed357beae9"). InnerVolumeSpecName "kube-api-access-s8tbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:11.411162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.411117 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:08:11.494099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.494060 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5691598-7eb7-449d-a4cf-b905f1db9cd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5691598-7eb7-449d-a4cf-b905f1db9cd5-plugins-config-volume\") pod \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " Oct 30 01:08:11.494312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.494255 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5691598-7eb7-449d-a4cf-b905f1db9cd5-kube-api-access-4k9fj\" (UniqueName: \"kubernetes.io/projected/c5691598-7eb7-449d-a4cf-b905f1db9cd5-kube-api-access-4k9fj\") pod \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\" (UID: \"c5691598-7eb7-449d-a4cf-b905f1db9cd5\") " Oct 30 01:08:11.494668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.494636 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5691598-7eb7-449d-a4cf-b905f1db9cd5-plugins-config-volume" pod "c5691598-7eb7-449d-a4cf-b905f1db9cd5" (UID: "c5691598-7eb7-449d-a4cf-b905f1db9cd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:11.495122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.495100 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5691598-7eb7-449d-a4cf-b905f1db9cd5-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:11.495174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.495131 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8tbk\" (UniqueName: \"kubernetes.io/projected/f5a9ee6a-f443-44b4-b8a9-32ed357beae9-kube-api-access-s8tbk\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:11.496735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.496704 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5691598-7eb7-449d-a4cf-b905f1db9cd5-kube-api-access-4k9fj" pod "c5691598-7eb7-449d-a4cf-b905f1db9cd5" (UID: "c5691598-7eb7-449d-a4cf-b905f1db9cd5"). InnerVolumeSpecName "kube-api-access-4k9fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:11.597127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.597036 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k9fj\" (UniqueName: \"kubernetes.io/projected/c5691598-7eb7-449d-a4cf-b905f1db9cd5-kube-api-access-4k9fj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:11.627460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.627422 4905 generic.go:356] "Generic (PLEG): container finished" podID="c5691598-7eb7-449d-a4cf-b905f1db9cd5" containerID="5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8" exitCode=0 Oct 30 01:08:11.627667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.627487 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" Oct 30 01:08:11.627667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.627504 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" event={"ID":"c5691598-7eb7-449d-a4cf-b905f1db9cd5","Type":"ContainerDied","Data":"5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8"} Oct 30 01:08:11.627667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.627570 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q" event={"ID":"c5691598-7eb7-449d-a4cf-b905f1db9cd5","Type":"ContainerDied","Data":"3860720456d5b21960882d859206a7d3e528e9612ca6812879609a3a26aa988c"} Oct 30 01:08:11.627667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.627593 4905 scope.go:122] "RemoveContainer" containerID="5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8" Oct 30 01:08:11.629618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.629371 4905 generic.go:356] "Generic (PLEG): container finished" podID="f5a9ee6a-f443-44b4-b8a9-32ed357beae9" containerID="fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963" exitCode=0 Oct 30 01:08:11.629618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.629421 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" Oct 30 01:08:11.629618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.629422 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" event={"ID":"f5a9ee6a-f443-44b4-b8a9-32ed357beae9","Type":"ContainerDied","Data":"fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963"} Oct 30 01:08:11.629618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.629598 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx" event={"ID":"f5a9ee6a-f443-44b4-b8a9-32ed357beae9","Type":"ContainerDied","Data":"44d294e45fbeab9a6054f8994e5cb2d2c2c123797c83368a2e010b8a1a076ac8"} Oct 30 01:08:11.640221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.640199 4905 scope.go:122] "RemoveContainer" containerID="5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8" Oct 30 01:08:11.640683 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:08:11.640650 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8\": container with ID starting with 5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8 not found: ID does not exist" containerID="5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8" Oct 30 01:08:11.640803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.640692 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8"} err="failed to get container status \"5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8\": rpc error: code = NotFound desc = could not find container \"5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8\": container with ID starting with 5eeb1c4916a30f92df8a1483f48aa5657b93f63fa57583881fcb5c94a00da9e8 not found: ID does not exist" Oct 30 01:08:11.640803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.640719 4905 scope.go:122] "RemoveContainer" containerID="fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963" Oct 30 01:08:11.654421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.654393 4905 scope.go:122] "RemoveContainer" containerID="fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963" Oct 30 01:08:11.654843 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:08:11.654818 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963\": container with ID starting with fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963 not found: ID does not exist" containerID="fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963" Oct 30 01:08:11.654931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.654855 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963"} err="failed to get container status \"fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963\": rpc error: code = NotFound desc = could not find container \"fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963\": container with ID starting with fbd2e9ac2fb2e4cec6f03440b6a67097911aa6807f0155648ea133988c1b1963 not found: ID does not exist" Oct 30 01:08:11.684365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.684325 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q"] Oct 30 01:08:11.696109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.691285 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9cc5-67886bf8c-vcg4q"] Oct 30 01:08:11.730031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.729996 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx"] Oct 30 01:08:11.734479 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:11.734449 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd9cc5-6754bcc7fb-8x5wx"] Oct 30 01:08:12.418770 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:08:12.418685 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:08:13.196230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:13.196189 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5691598-7eb7-449d-a4cf-b905f1db9cd5" path="/var/lib/kubelet/pods/c5691598-7eb7-449d-a4cf-b905f1db9cd5/volumes" Oct 30 01:08:13.196772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:13.196755 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a9ee6a-f443-44b4-b8a9-32ed357beae9" path="/var/lib/kubelet/pods/f5a9ee6a-f443-44b4-b8a9-32ed357beae9/volumes" Oct 30 01:08:16.930502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.930459 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt"] Oct 30 01:08:16.931650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931625 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" Oct 30 01:08:16.931650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931648 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" Oct 30 01:08:16.931843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931722 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" Oct 30 01:08:16.931843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931731 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" Oct 30 01:08:16.931843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931763 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f5a9ee6a-f443-44b4-b8a9-32ed357beae9" containerName="payload-processing" Oct 30 01:08:16.931843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931770 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a9ee6a-f443-44b4-b8a9-32ed357beae9" containerName="payload-processing" Oct 30 01:08:16.931843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931791 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c5691598-7eb7-449d-a4cf-b905f1db9cd5" containerName="payload-pre-processing" Oct 30 01:08:16.931843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.931799 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5691598-7eb7-449d-a4cf-b905f1db9cd5" containerName="payload-pre-processing" Oct 30 01:08:16.932145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.932029 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" Oct 30 01:08:16.932145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.932057 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffa3510-3a48-4aff-b299-584ce5e7a0ea" containerName="istio-proxy" Oct 30 01:08:16.932145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.932071 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a9ee6a-f443-44b4-b8a9-32ed357beae9" containerName="payload-processing" Oct 30 01:08:16.932145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.932084 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5691598-7eb7-449d-a4cf-b905f1db9cd5" containerName="payload-pre-processing" Oct 30 01:08:16.937730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.937703 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:16.957396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.957366 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt"] Oct 30 01:08:16.962307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:16.962279 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bc6092-serving-cert\"" Oct 30 01:08:17.031930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.031889 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-metrics-certs\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.032129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.031965 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgf7w\" (UniqueName: \"kubernetes.io/projected/9250ee6e-2572-4c59-a49e-24af53cd813a-kube-api-access-mgf7w\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.032181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.032167 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9250ee6e-2572-4c59-a49e-24af53cd813a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.032246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.032227 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-maas-api-tls\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.146549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.145924 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9250ee6e-2572-4c59-a49e-24af53cd813a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.146549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.146027 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-maas-api-tls\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.146549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.146149 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-metrics-certs\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.146549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.146192 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgf7w\" (UniqueName: \"kubernetes.io/projected/9250ee6e-2572-4c59-a49e-24af53cd813a-kube-api-access-mgf7w\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.161535 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.161111 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-metrics-certs\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.170547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.168065 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9250ee6e-2572-4c59-a49e-24af53cd813a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.185615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.179651 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-maas-api-tls\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.207543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.202928 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgf7w\" (UniqueName: \"kubernetes.io/projected/9250ee6e-2572-4c59-a49e-24af53cd813a-kube-api-access-mgf7w\") pod \"maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.269546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.268834 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:17.591166 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:08:17.591124 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9250ee6e_2572_4c59_a49e_24af53cd813a.slice/crio-27530e7969350483c0ca2806a13c54f96f867d4fdea1b6c88464bac36d5cb16e WatchSource:0}: Error finding container 27530e7969350483c0ca2806a13c54f96f867d4fdea1b6c88464bac36d5cb16e: Status 404 returned error can't find the container with id 27530e7969350483c0ca2806a13c54f96f867d4fdea1b6c88464bac36d5cb16e Oct 30 01:08:17.591897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.591872 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt"] Oct 30 01:08:17.755604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:17.753383 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" event={"ID":"9250ee6e-2572-4c59-a49e-24af53cd813a","Type":"ContainerStarted","Data":"27530e7969350483c0ca2806a13c54f96f867d4fdea1b6c88464bac36d5cb16e"} Oct 30 01:08:18.767996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:18.767952 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" event={"ID":"9250ee6e-2572-4c59-a49e-24af53cd813a","Type":"ContainerStarted","Data":"ea2be39dcb73132a09717370c3421eb5abf3afec17c9ba5ed8552ca80112b0e0"} Oct 30 01:08:18.768460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:18.768037 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:18.836625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:18.831712 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" podStartSLOduration=2.44303992 podStartE2EDuration="2.831689067s" totalImagesPullingTime="388.649147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:18.811503991 +0000 UTC m=+1783.754677183" watchObservedRunningTime="2026-10-30 01:08:18.831689067 +0000 UTC m=+1783.774862263" Oct 30 01:08:24.784258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:24.784223 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:08:25.935837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:25.935782 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd"] Oct 30 01:08:25.940945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:25.940913 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:25.963790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:25.963758 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd"] Oct 30 01:08:25.970998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:25.970950 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-d0ae0d-serving-cert\"" Oct 30 01:08:26.001822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.001733 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hbm\" (UniqueName: \"kubernetes.io/projected/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-kube-api-access-z5hbm\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.002194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.001895 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.002194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.002029 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.002454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.002391 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.104335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.104290 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.104576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.104474 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.104732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.104711 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hbm\" (UniqueName: \"kubernetes.io/projected/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-kube-api-access-z5hbm\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.104792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.104777 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.105491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.105467 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.107195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.107171 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.107429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.107402 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.125945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.125912 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hbm\" (UniqueName: \"kubernetes.io/projected/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-kube-api-access-z5hbm\") pod \"maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.259004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.258965 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:26.611481 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:08:26.611447 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2b4abf_aafe_4c3f_a52d_d05862efb7ea.slice/crio-509ca50f1423bf4162403f6d22eb929e4921520d04441bc7e457b29a8b2b5d3e WatchSource:0}: Error finding container 509ca50f1423bf4162403f6d22eb929e4921520d04441bc7e457b29a8b2b5d3e: Status 404 returned error can't find the container with id 509ca50f1423bf4162403f6d22eb929e4921520d04441bc7e457b29a8b2b5d3e Oct 30 01:08:26.615187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.614074 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd"] Oct 30 01:08:26.922796 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:26.922694 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" event={"ID":"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea","Type":"ContainerStarted","Data":"509ca50f1423bf4162403f6d22eb929e4921520d04441bc7e457b29a8b2b5d3e"} Oct 30 01:08:27.935073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:27.935037 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" event={"ID":"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea","Type":"ContainerStarted","Data":"4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4"} Oct 30 01:08:27.935620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:27.935146 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:27.984654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:27.984590 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" podStartSLOduration=2.566218087 podStartE2EDuration="2.984572893s" totalImagesPullingTime="418.354806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:27.980184823 +0000 UTC m=+1792.923358019" watchObservedRunningTime="2026-10-30 01:08:27.984572893 +0000 UTC m=+1792.927746085" Oct 30 01:08:33.955962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:33.955930 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:08:34.187042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:34.187008 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd"] Oct 30 01:08:34.187247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:34.187224 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" containerID="cri-o://52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b" gracePeriod=30 Oct 30 01:08:35.732687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:35.732656 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:08:35.735953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:35.735690 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:08:39.583496 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.583462 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:08:39.643875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.643838 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-certs\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.644241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644220 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-kube-api-access-4c7q9\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-kube-api-access-4c7q9\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.644375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644361 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-credential-socket\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.644551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644505 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-data\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.644694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644583 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-envoy\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.644694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644655 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istiod-ca-cert\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.644807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644768 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-socket\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644865 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-token\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.644895 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-podinfo\") pod \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\" (UID: \"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47\") " Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.645337 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-credential-socket" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646117 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istiod-ca-cert" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.645082 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-data" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646504 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-socket" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646638 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-certs" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646899 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646922 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646937 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646949 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.646999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.646974 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.650166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.650138 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-envoy" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:39.650348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.650182 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-podinfo" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:39.652894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.652809 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-token" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:39.654002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.653966 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-kube-api-access-4c7q9" pod "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" (UID: "3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47"). InnerVolumeSpecName "kube-api-access-4c7q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:39.749126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.749086 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c7q9\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-kube-api-access-4c7q9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.749126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.749123 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.749356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.749136 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:39.749356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:39.749147 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:08:40.071985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.071944 4905 generic.go:356] "Generic (PLEG): container finished" podID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerID="52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b" exitCode=0 Oct 30 01:08:40.072280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.072012 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" Oct 30 01:08:40.072280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.072031 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" event={"ID":"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47","Type":"ContainerDied","Data":"52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b"} Oct 30 01:08:40.072280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.072134 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd" event={"ID":"3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47","Type":"ContainerDied","Data":"ab276e351fc7b178d849ea6af4768dc199996b0c53636accda2e2cb59f44ac74"} Oct 30 01:08:40.072280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.072164 4905 scope.go:122] "RemoveContainer" containerID="52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b" Oct 30 01:08:40.087276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.087251 4905 scope.go:122] "RemoveContainer" containerID="52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b" Oct 30 01:08:40.087767 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:08:40.087739 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b\": container with ID starting with 52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b not found: ID does not exist" containerID="52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b" Oct 30 01:08:40.087857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.087775 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b"} err="failed to get container status \"52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b\": rpc error: code = NotFound desc = could not find container \"52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b\": container with ID starting with 52bfbcc9d512d73a7d34a2da9faa013b1fe3458a73f50062f8c026b80bb6ca1b not found: ID does not exist" Oct 30 01:08:40.133330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.133290 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd"] Oct 30 01:08:40.144435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:40.144403 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vglwd"] Oct 30 01:08:41.195130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:41.195096 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" path="/var/lib/kubelet/pods/3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47/volumes" Oct 30 01:08:42.421183 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:08:42.421148 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:08:43.389608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:43.389578 4905 scope.go:122] "RemoveContainer" containerID="1924fc15f195197c810eaf7ad58957087e26b5e8ef98a0128726da9e9c56b4d9" Oct 30 01:08:43.402993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:43.402964 4905 scope.go:122] "RemoveContainer" containerID="dc4fce2fad9b903a0f004a3403a4e65ccda78532d13b5145d83d5ca7d294c214" Oct 30 01:08:43.414609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:08:43.414587 4905 scope.go:122] "RemoveContainer" containerID="94c0d9d662cd2dae8dada90082ac35bac3322d320e1f7451064c32bcfa85e700" Oct 30 01:09:12.601719 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:09:12.598773 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:09:16.672186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:16.671827 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:09:16.672186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:16.671904 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:21.750557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:21.749165 4905 generic.go:356] "Generic (PLEG): container finished" podID="f280e47e-36f4-4451-a458-311662858ecd" containerID="13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd" exitCode=0 Oct 30 01:09:21.750557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:21.749317 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" event={"ID":"f280e47e-36f4-4451-a458-311662858ecd","Type":"ContainerDied","Data":"13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd"} Oct 30 01:09:21.750557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:21.749909 4905 scope.go:122] "RemoveContainer" containerID="13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd" Oct 30 01:09:21.754939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:21.754048 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:09:21.754939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:21.754143 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:09:21.754939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:21.754187 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:09:22.581094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.579793 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": read tcp 10.132.0.2:44144->10.132.0.238:15021: read: connection reset by peer" start-of-body= Oct 30 01:09:22.581094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.579867 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": read tcp 10.132.0.2:44144->10.132.0.238:15021: read: connection reset by peer" Oct 30 01:09:22.763676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.763585 4905 generic.go:356] "Generic (PLEG): container finished" podID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerID="f474304491c598d8f17f6176621622605723fd495ae5aceb845b69ea8be1f0f1" exitCode=0 Oct 30 01:09:22.764175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.763693 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" event={"ID":"70491b5d-60ce-4630-997e-9f16b05bc8b4","Type":"ContainerDied","Data":"f474304491c598d8f17f6176621622605723fd495ae5aceb845b69ea8be1f0f1"} Oct 30 01:09:22.764375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.764347 4905 scope.go:122] "RemoveContainer" containerID="f474304491c598d8f17f6176621622605723fd495ae5aceb845b69ea8be1f0f1" Oct 30 01:09:22.766629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.766607 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" event={"ID":"f280e47e-36f4-4451-a458-311662858ecd","Type":"ContainerStarted","Data":"148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1"} Oct 30 01:09:22.766763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.766695 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:09:22.766834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.766769 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:09:22.766834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:22.766801 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:09:23.621540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:23.621484 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:09:23.621930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:23.621825 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 01:09:23.622066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:23.621954 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 01:09:23.872504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:23.872407 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" event={"ID":"70491b5d-60ce-4630-997e-9f16b05bc8b4","Type":"ContainerStarted","Data":"2c87d657eeb30879083c1231814cb6a9ff9952a47827af8fdffa072bc747d921"} Oct 30 01:09:24.622245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:24.622201 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 01:09:24.622527 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:24.622485 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 01:09:24.687591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:24.687504 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:09:24.687891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:24.687875 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:09:24.688012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:24.687996 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 01:09:25.627545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:25.622708 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 01:09:25.627545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:25.622769 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 01:09:25.688182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:25.688144 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:09:25.688502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:25.688474 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 01:09:26.622134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:26.622098 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 01:09:26.622296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:26.622163 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 01:09:26.687548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:26.687415 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:09:26.687548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:26.687474 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 01:09:27.622106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:27.622076 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 01:09:27.622354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:27.622324 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 01:09:27.689031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:27.688990 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:09:27.690104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:27.689694 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 01:09:28.623540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:28.623011 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 01:09:28.623540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:28.623073 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 01:09:28.687510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:28.687469 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:09:28.687703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:28.687555 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 01:09:29.640919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:29.640861 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:09:29.643098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:29.643076 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:09:29.653428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:29.653403 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:09:29.687456 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:29.687421 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:09:29.687599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:29.687465 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 01:09:30.213483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:30.213444 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs"] Oct 30 01:09:30.213741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:30.213711 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" podUID="7a99d858-b126-4780-9d53-2004398b2da9" containerName="maas-api" containerID="cri-o://0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403" gracePeriod=30 Oct 30 01:09:30.750021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:30.749841 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:09:30.753403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:30.751469 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:09:30.756906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:30.755726 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:09:30.952542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:30.941313 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:09:31.055592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.054804 4905 generic.go:356] "Generic (PLEG): container finished" podID="7a99d858-b126-4780-9d53-2004398b2da9" containerID="0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403" exitCode=0 Oct 30 01:09:31.056078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.056037 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" Oct 30 01:09:31.056689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.056649 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" event={"ID":"7a99d858-b126-4780-9d53-2004398b2da9","Type":"ContainerDied","Data":"0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403"} Oct 30 01:09:31.056907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.056871 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs" event={"ID":"7a99d858-b126-4780-9d53-2004398b2da9","Type":"ContainerDied","Data":"d86bd1fdcd9c35011d9ab59208c6cd5ecd90c9503acc326fa8a2e6353c8c3a67"} Oct 30 01:09:31.057105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.057074 4905 scope.go:122] "RemoveContainer" containerID="0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403" Oct 30 01:09:31.102692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.102658 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-metrics-certs\") pod \"7a99d858-b126-4780-9d53-2004398b2da9\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " Oct 30 01:09:31.102845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.102808 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a99d858-b126-4780-9d53-2004398b2da9-kube-api-access-9pcxm\" (UniqueName: \"kubernetes.io/projected/7a99d858-b126-4780-9d53-2004398b2da9-kube-api-access-9pcxm\") pod \"7a99d858-b126-4780-9d53-2004398b2da9\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " Oct 30 01:09:31.102909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.102892 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-maas-api-tls\") pod \"7a99d858-b126-4780-9d53-2004398b2da9\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " Oct 30 01:09:31.103026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.103008 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a99d858-b126-4780-9d53-2004398b2da9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a99d858-b126-4780-9d53-2004398b2da9-openshift-service-ca\") pod \"7a99d858-b126-4780-9d53-2004398b2da9\" (UID: \"7a99d858-b126-4780-9d53-2004398b2da9\") " Oct 30 01:09:31.103669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.103623 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a99d858-b126-4780-9d53-2004398b2da9-openshift-service-ca" pod "7a99d858-b126-4780-9d53-2004398b2da9" (UID: "7a99d858-b126-4780-9d53-2004398b2da9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:31.103825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.103802 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a99d858-b126-4780-9d53-2004398b2da9-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:31.105784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.105743 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-metrics-certs" pod "7a99d858-b126-4780-9d53-2004398b2da9" (UID: "7a99d858-b126-4780-9d53-2004398b2da9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:31.106316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.106241 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-maas-api-tls" pod "7a99d858-b126-4780-9d53-2004398b2da9" (UID: "7a99d858-b126-4780-9d53-2004398b2da9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:31.106316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.106297 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a99d858-b126-4780-9d53-2004398b2da9-kube-api-access-9pcxm" pod "7a99d858-b126-4780-9d53-2004398b2da9" (UID: "7a99d858-b126-4780-9d53-2004398b2da9"). InnerVolumeSpecName "kube-api-access-9pcxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:31.107880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.107856 4905 scope.go:122] "RemoveContainer" containerID="0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403" Oct 30 01:09:31.108655 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:09:31.108628 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403\": container with ID starting with 0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403 not found: ID does not exist" containerID="0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403" Oct 30 01:09:31.108768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.108665 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403"} err="failed to get container status \"0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403\": rpc error: code = NotFound desc = could not find container \"0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403\": container with ID starting with 0f860616402cf3ecefb70edc66e4313a2a750ee022f5f4a37bc4f15f78081403 not found: ID does not exist" Oct 30 01:09:31.206232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.206190 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:31.206232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.206221 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pcxm\" (UniqueName: \"kubernetes.io/projected/7a99d858-b126-4780-9d53-2004398b2da9-kube-api-access-9pcxm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:31.206232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.206234 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a99d858-b126-4780-9d53-2004398b2da9-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:31.410708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.410603 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs"] Oct 30 01:09:31.422111 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:31.422074 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9cc5-96f6f75b5-ngfrs"] Oct 30 01:09:33.194174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:33.194126 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a99d858-b126-4780-9d53-2004398b2da9" path="/var/lib/kubelet/pods/7a99d858-b126-4780-9d53-2004398b2da9/volumes" Oct 30 01:09:38.582207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:38.582153 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m"] Oct 30 01:09:38.582811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:38.582466 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" containerID="cri-o://026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf" gracePeriod=30 Oct 30 01:09:42.431396 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:09:42.431349 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:09:43.500354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:43.498835 4905 scope.go:122] "RemoveContainer" containerID="59e1568a0a0826eef12877c67cc7813425d5a6fbe2358c6b0c9e1675b0b1b456" Oct 30 01:09:43.526786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:43.526748 4905 scope.go:122] "RemoveContainer" containerID="f09d3b1892b44e0934c270366802b53927ca2351e7d9a21714acd65fac63627c" Oct 30 01:09:43.594005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:43.593851 4905 scope.go:122] "RemoveContainer" containerID="cada77802c7375e81aecdd29fe4acca0f1991c77bbdae1bd99567fe1aa6fb404" Oct 30 01:09:43.980566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:43.980534 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:09:44.119652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.119397 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-data\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.119652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.119503 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-envoy\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.119730 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istiod-ca-cert\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.119795 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-certs\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.119844 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-podinfo\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.119953 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-credential-socket\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120039 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-kube-api-access-9cjpt\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-kube-api-access-9cjpt\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120105 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-socket\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120162 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-certs" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.120820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120185 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-token\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-token\") pod \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\" (UID: \"02bb569b-2dc4-48ec-9d22-6b04e2d0f088\") " Oct 30 01:09:44.120820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120412 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-socket" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.120820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120592 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-credential-socket" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.120820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120704 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istiod-ca-cert" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:44.121267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.120979 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-data" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.121448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.121365 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.121448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.121389 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.121448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.121405 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.121448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.121418 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.121448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.121432 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.124129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.124098 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-kube-api-access-9cjpt" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "kube-api-access-9cjpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:44.124226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.124198 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-envoy" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.131147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.131078 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-token" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:44.131265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.131145 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-podinfo" pod "02bb569b-2dc4-48ec-9d22-6b04e2d0f088" (UID: "02bb569b-2dc4-48ec-9d22-6b04e2d0f088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:44.222902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.222859 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.222902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.222895 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.222902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.222908 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.223162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.222921 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjpt\" (UniqueName: \"kubernetes.io/projected/02bb569b-2dc4-48ec-9d22-6b04e2d0f088-kube-api-access-9cjpt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.447749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.445924 4905 generic.go:356] "Generic (PLEG): container finished" podID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerID="026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf" exitCode=0 Oct 30 01:09:44.447749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.446122 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" Oct 30 01:09:44.448064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.446145 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" event={"ID":"02bb569b-2dc4-48ec-9d22-6b04e2d0f088","Type":"ContainerDied","Data":"026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf"} Oct 30 01:09:44.448131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.448105 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m" event={"ID":"02bb569b-2dc4-48ec-9d22-6b04e2d0f088","Type":"ContainerDied","Data":"0bc5cdd8ec71cd7d55a404b9f10ef7f6fa25e725d9a3ea4113e0f07f339f0cad"} Oct 30 01:09:44.448191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.448142 4905 scope.go:122] "RemoveContainer" containerID="026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf" Oct 30 01:09:44.465213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.465175 4905 scope.go:122] "RemoveContainer" containerID="a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e" Oct 30 01:09:44.480753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.480723 4905 scope.go:122] "RemoveContainer" containerID="026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf" Oct 30 01:09:44.482665 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:09:44.482638 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf\": container with ID starting with 026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf not found: ID does not exist" containerID="026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf" Oct 30 01:09:44.482765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.482671 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf"} err="failed to get container status \"026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf\": rpc error: code = NotFound desc = could not find container \"026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf\": container with ID starting with 026cba58ca8d3a0f391ad1ee50189233758e0cf7bd0b36946385829bd63d4faf not found: ID does not exist" Oct 30 01:09:44.482765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.482691 4905 scope.go:122] "RemoveContainer" containerID="a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e" Oct 30 01:09:44.483040 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:09:44.483020 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e\": container with ID starting with a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e not found: ID does not exist" containerID="a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e" Oct 30 01:09:44.483101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.483047 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e"} err="failed to get container status \"a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e\": rpc error: code = NotFound desc = could not find container \"a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e\": container with ID starting with a063be64705a113a26376564933f51236a34da292f0dd1849f1cfda41bc8973e not found: ID does not exist" Oct 30 01:09:44.507772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.507745 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m"] Oct 30 01:09:44.512330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:44.512304 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bd9cc5-openshift-default-67477c6bb7-g4h9m"] Oct 30 01:09:44.544749 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:09:44.544713 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bb569b_2dc4_48ec_9d22_6b04e2d0f088.slice/crio-0bc5cdd8ec71cd7d55a404b9f10ef7f6fa25e725d9a3ea4113e0f07f339f0cad\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bb569b_2dc4_48ec_9d22_6b04e2d0f088.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:09:45.193555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.193481 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" path="/var/lib/kubelet/pods/02bb569b-2dc4-48ec-9d22-6b04e2d0f088/volumes" Oct 30 01:09:45.729240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.729198 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr"] Oct 30 01:09:45.730647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730618 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" Oct 30 01:09:45.730647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730644 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" Oct 30 01:09:45.730932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730707 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7a99d858-b126-4780-9d53-2004398b2da9" containerName="maas-api" Oct 30 01:09:45.730932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730715 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a99d858-b126-4780-9d53-2004398b2da9" containerName="maas-api" Oct 30 01:09:45.730932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730728 4905 cpu_manager.go:414] "containerMap: removing container" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" Oct 30 01:09:45.730932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730736 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" Oct 30 01:09:45.730932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730747 4905 cpu_manager.go:414] "containerMap: removing container" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" Oct 30 01:09:45.730932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730756 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" Oct 30 01:09:45.731224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730965 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a99d858-b126-4780-9d53-2004398b2da9" containerName="maas-api" Oct 30 01:09:45.731224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730980 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" Oct 30 01:09:45.731224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.730991 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bb569b-2dc4-48ec-9d22-6b04e2d0f088" containerName="istio-proxy" Oct 30 01:09:45.731224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.731011 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbb0d9f-b739-4ace-90cb-81a4ebc6cd47" containerName="istio-proxy" Oct 30 01:09:45.737473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.737433 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:45.755350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.755317 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr"] Oct 30 01:09:45.759874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:45.759291 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b312f7\"" Oct 30 01:09:46.030545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.024896 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkcl\" (UniqueName: \"kubernetes.io/projected/fd2d942e-d568-4861-bf79-bd8799145eda-kube-api-access-gzkcl\") pod \"payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:46.030545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.024992 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2d942e-d568-4861-bf79-bd8799145eda-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:46.146552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.146047 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkcl\" (UniqueName: \"kubernetes.io/projected/fd2d942e-d568-4861-bf79-bd8799145eda-kube-api-access-gzkcl\") pod \"payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:46.146552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.146129 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2d942e-d568-4861-bf79-bd8799145eda-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:46.150543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.147256 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2d942e-d568-4861-bf79-bd8799145eda-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:46.174127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.173543 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkcl\" (UniqueName: \"kubernetes.io/projected/fd2d942e-d568-4861-bf79-bd8799145eda-kube-api-access-gzkcl\") pod \"payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:46.370448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.370345 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:46.687558 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:09:46.686574 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2d942e_d568_4861_bf79_bd8799145eda.slice/crio-3f6312adb6ece0aac482602322c225c35afdab6e4132093c85ada20f4b42f6af WatchSource:0}: Error finding container 3f6312adb6ece0aac482602322c225c35afdab6e4132093c85ada20f4b42f6af: Status 404 returned error can't find the container with id 3f6312adb6ece0aac482602322c225c35afdab6e4132093c85ada20f4b42f6af Oct 30 01:09:46.695498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:46.695464 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr"] Oct 30 01:09:47.495729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:47.495684 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" event={"ID":"fd2d942e-d568-4861-bf79-bd8799145eda","Type":"ContainerStarted","Data":"4341babe456c8ab8a344c715035ed66a45665d5c3691fdabc13f8d3bb06407d3"} Oct 30 01:09:47.496265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:47.495754 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" event={"ID":"fd2d942e-d568-4861-bf79-bd8799145eda","Type":"ContainerStarted","Data":"3f6312adb6ece0aac482602322c225c35afdab6e4132093c85ada20f4b42f6af"} Oct 30 01:09:47.496265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:47.495781 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:09:47.541488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:47.541409 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" podStartSLOduration=2.541385534 podStartE2EDuration="2.541385534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:47.540268973 +0000 UTC m=+1872.483442166" watchObservedRunningTime="2026-10-30 01:09:47.541385534 +0000 UTC m=+1872.484558729" Oct 30 01:09:55.884551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:55.862629 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:55.912543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:55.896779 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:55.982163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:55.938265 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:56.065661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.058790 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:56.291546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.251197 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:56.429925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.429878 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:56.815545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.812255 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:56.836544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.832668 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:56.891469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.888666 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:56.892391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.892363 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:09:56.951647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:56.951028 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:09:58.521938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:09:58.521907 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:10:12.418266 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:10:12.418230 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:10:24.355264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.355216 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n"] Oct 30 01:10:24.361779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.361751 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.382766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.382736 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n"] Oct 30 01:10:24.392092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.391254 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2bf8a-openshift-default-dockercfg-cf8qf\"" Oct 30 01:10:24.447077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447030 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-certs\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447170 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-podinfo\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447228 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7f97\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-kube-api-access-q7f97\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447369 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istiod-ca-cert\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447612 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447677 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-token\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447710 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-credential-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447847 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-data\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.447945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.447916 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-envoy\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549345 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-envoy\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549403 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-certs\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549441 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-podinfo\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549479 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7f97\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-kube-api-access-q7f97\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549536 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istiod-ca-cert\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549711 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549770 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-token\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.549900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549794 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-credential-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.550082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.549907 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-data\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.550169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.550130 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.550237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.550180 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-certs\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.550453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.550430 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-data\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.550683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.550495 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-credential-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.550683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.550582 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istiod-ca-cert\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.552485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.552461 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-envoy\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.552594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.552500 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-podinfo\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.570314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.570283 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-token\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.570608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.570581 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7f97\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-kube-api-access-q7f97\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:24.679840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:24.679733 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:25.087189 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:10:25.087149 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8bf617_86cd_4e4f_ac1b_a67508dcd1c7.slice/crio-63c8a766e040ee0eb22ae8fda52bf72bb6ecb0dcccbfaf4425188d5daeaa59d6 WatchSource:0}: Error finding container 63c8a766e040ee0eb22ae8fda52bf72bb6ecb0dcccbfaf4425188d5daeaa59d6: Status 404 returned error can't find the container with id 63c8a766e040ee0eb22ae8fda52bf72bb6ecb0dcccbfaf4425188d5daeaa59d6 Oct 30 01:10:25.089610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.089542 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n"] Oct 30 01:10:25.091606 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.091561 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:10:25.091719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.091656 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:10:25.091719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.091703 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:10:25.445688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.445583 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" event={"ID":"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7","Type":"ContainerStarted","Data":"343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80"} Oct 30 01:10:25.445688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.445643 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" event={"ID":"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7","Type":"ContainerStarted","Data":"63c8a766e040ee0eb22ae8fda52bf72bb6ecb0dcccbfaf4425188d5daeaa59d6"} Oct 30 01:10:25.492169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.492102 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podStartSLOduration=1.492085777 podStartE2EDuration="1.492085777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:25.48860081 +0000 UTC m=+1910.431774002" watchObservedRunningTime="2026-10-30 01:10:25.492085777 +0000 UTC m=+1910.435258970" Oct 30 01:10:25.682574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:25.682228 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:26.682395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:26.682356 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 01:10:26.682881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:26.682406 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 01:10:27.681160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.681120 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 01:10:27.681344 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.681186 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 01:10:27.954305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.954219 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6"] Oct 30 01:10:27.962382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.962354 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:27.965632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.964723 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6"] Oct 30 01:10:27.975548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.971643 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrnh\" (UniqueName: \"kubernetes.io/projected/9f65e2dc-800e-4f18-8961-2b05692cd66a-kube-api-access-xnrnh\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:27.975548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.971776 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:27.975548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.972087 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-metrics-certs\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:27.975548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.972141 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f65e2dc-800e-4f18-8961-2b05692cd66a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:27.981504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:27.981484 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b312f7-serving-cert\"" Oct 30 01:10:28.074124 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.074089 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.074335 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:10:28.074245 4905 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-serving-cert: secret "maas-api-e2e-inf-a-b312f7-serving-cert" not found Oct 30 01:10:28.074335 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:10:28.074311 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls podName:9f65e2dc-800e-4f18-8961-2b05692cd66a nodeName:}" failed. No retries permitted until 2026-10-30 01:10:28.574296018 +0000 UTC m=+1913.517469201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls") pod "maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" (UID: "9f65e2dc-800e-4f18-8961-2b05692cd66a") : secret "maas-api-e2e-inf-a-b312f7-serving-cert" not found Oct 30 01:10:28.074468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.074376 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-metrics-certs\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.074468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.074412 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f65e2dc-800e-4f18-8961-2b05692cd66a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.074605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.074490 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrnh\" (UniqueName: \"kubernetes.io/projected/9f65e2dc-800e-4f18-8961-2b05692cd66a-kube-api-access-xnrnh\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.075168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.075136 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f65e2dc-800e-4f18-8961-2b05692cd66a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.077178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.077156 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-metrics-certs\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.097531 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.097504 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrnh\" (UniqueName: \"kubernetes.io/projected/9f65e2dc-800e-4f18-8961-2b05692cd66a-kube-api-access-xnrnh\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.587353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.587312 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.589935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.589915 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls\") pod \"maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:28.681104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.681072 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 01:10:28.681302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.681134 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 01:10:28.894185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:28.890888 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:29.304558 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:10:29.301924 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f65e2dc_800e_4f18_8961_2b05692cd66a.slice/crio-6340b9a069a3be049e13ab6d2f9392cfd8d0e857897bacf07191f29651ce7f6e WatchSource:0}: Error finding container 6340b9a069a3be049e13ab6d2f9392cfd8d0e857897bacf07191f29651ce7f6e: Status 404 returned error can't find the container with id 6340b9a069a3be049e13ab6d2f9392cfd8d0e857897bacf07191f29651ce7f6e Oct 30 01:10:29.304558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:29.304039 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:10:29.304558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:29.304226 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6"] Oct 30 01:10:29.520434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:29.520388 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" event={"ID":"9f65e2dc-800e-4f18-8961-2b05692cd66a","Type":"ContainerStarted","Data":"6340b9a069a3be049e13ab6d2f9392cfd8d0e857897bacf07191f29651ce7f6e"} Oct 30 01:10:29.681604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:29.680651 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 01:10:29.681604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:29.680710 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 01:10:30.575367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:30.575320 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" event={"ID":"9f65e2dc-800e-4f18-8961-2b05692cd66a","Type":"ContainerStarted","Data":"9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b"} Oct 30 01:10:30.576319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:30.576290 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:30.623250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:30.623187 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" podStartSLOduration=3.213702061 podStartE2EDuration="3.623174248s" totalImagesPullingTime="409.472187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:30.621842791 +0000 UTC m=+1915.565015984" watchObservedRunningTime="2026-10-30 01:10:30.623174248 +0000 UTC m=+1915.566347441" Oct 30 01:10:30.680360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:30.680308 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 01:10:30.680599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:30.680372 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 01:10:30.907070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:30.906977 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25"] Oct 30 01:10:30.964780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:30.964084 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9"] Oct 30 01:10:31.683579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:31.680916 4905 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 01:10:31.683579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:31.680975 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 01:10:32.685694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:32.685660 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:32.686176 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:32.686158 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:32.687094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:32.687073 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:10:35.907414 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:35.907295 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" podUID="7c94fbff-b58f-4434-a509-01f353a46658" containerName="payload-processing" containerID="cri-o://ad9362b620589eb61f4eea97305ee9cdaee5820e3f8ee1de626616ab1e0bb9a6" gracePeriod=25 Oct 30 01:10:35.964800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:35.964760 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" podUID="06211214-57e4-432e-b254-0b30fea59733" containerName="payload-pre-processing" containerID="cri-o://30b65753688a50a8b29d5cd1b39999911ac0cd0d0dd6f4910d88b88bf492f99d" gracePeriod=25 Oct 30 01:10:36.658661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:36.658630 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:10:37.075366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:37.075221 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" podUID="7c94fbff-b58f-4434-a509-01f353a46658" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:38.264923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:38.264870 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" podUID="06211214-57e4-432e-b254-0b30fea59733" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:42.416769 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:10:42.416725 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:10:43.781413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:43.781380 4905 scope.go:122] "RemoveContainer" containerID="6b275f602faf7ad4d23e41be79112de5342844b88014257a19d95772f77e4f56" Oct 30 01:10:43.792711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:43.792685 4905 scope.go:122] "RemoveContainer" containerID="8995d0e053ebc50138564adad20bdde942c2404a9f5f69a636b186687e926124" Oct 30 01:10:43.804650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:43.804616 4905 scope.go:122] "RemoveContainer" containerID="91126e3e8a2459b3a33208ae176f59b0b0bf0444dbf76b4a177085a0dcb01046" Oct 30 01:10:47.064240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:47.064141 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" podUID="7c94fbff-b58f-4434-a509-01f353a46658" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:48.265293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:48.265256 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" podUID="06211214-57e4-432e-b254-0b30fea59733" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:56.200847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.200761 4905 generic.go:356] "Generic (PLEG): container finished" podID="06211214-57e4-432e-b254-0b30fea59733" containerID="30b65753688a50a8b29d5cd1b39999911ac0cd0d0dd6f4910d88b88bf492f99d" exitCode=0 Oct 30 01:10:56.205426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.200853 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" event={"ID":"06211214-57e4-432e-b254-0b30fea59733","Type":"ContainerDied","Data":"30b65753688a50a8b29d5cd1b39999911ac0cd0d0dd6f4910d88b88bf492f99d"} Oct 30 01:10:56.205426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.203218 4905 generic.go:356] "Generic (PLEG): container finished" podID="7c94fbff-b58f-4434-a509-01f353a46658" containerID="ad9362b620589eb61f4eea97305ee9cdaee5820e3f8ee1de626616ab1e0bb9a6" exitCode=0 Oct 30 01:10:56.205426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.203337 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" event={"ID":"7c94fbff-b58f-4434-a509-01f353a46658","Type":"ContainerDied","Data":"ad9362b620589eb61f4eea97305ee9cdaee5820e3f8ee1de626616ab1e0bb9a6"} Oct 30 01:10:56.340782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.338358 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:10:56.444906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.444864 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c94fbff-b58f-4434-a509-01f353a46658-kube-api-access-2rj7x\" (UniqueName: \"kubernetes.io/projected/7c94fbff-b58f-4434-a509-01f353a46658-kube-api-access-2rj7x\") pod \"7c94fbff-b58f-4434-a509-01f353a46658\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " Oct 30 01:10:56.445606 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.445412 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c94fbff-b58f-4434-a509-01f353a46658-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94fbff-b58f-4434-a509-01f353a46658-plugins-config-volume\") pod \"7c94fbff-b58f-4434-a509-01f353a46658\" (UID: \"7c94fbff-b58f-4434-a509-01f353a46658\") " Oct 30 01:10:56.447447 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.447416 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c94fbff-b58f-4434-a509-01f353a46658-plugins-config-volume" pod "7c94fbff-b58f-4434-a509-01f353a46658" (UID: "7c94fbff-b58f-4434-a509-01f353a46658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:56.455868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.454319 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c94fbff-b58f-4434-a509-01f353a46658-kube-api-access-2rj7x" pod "7c94fbff-b58f-4434-a509-01f353a46658" (UID: "7c94fbff-b58f-4434-a509-01f353a46658"). InnerVolumeSpecName "kube-api-access-2rj7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:56.486574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.486542 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:10:56.548815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.548773 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06211214-57e4-432e-b254-0b30fea59733-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06211214-57e4-432e-b254-0b30fea59733-plugins-config-volume\") pod \"06211214-57e4-432e-b254-0b30fea59733\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " Oct 30 01:10:56.549016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.548925 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06211214-57e4-432e-b254-0b30fea59733-kube-api-access-j9cc2\" (UniqueName: \"kubernetes.io/projected/06211214-57e4-432e-b254-0b30fea59733-kube-api-access-j9cc2\") pod \"06211214-57e4-432e-b254-0b30fea59733\" (UID: \"06211214-57e4-432e-b254-0b30fea59733\") " Oct 30 01:10:56.549297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.549266 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06211214-57e4-432e-b254-0b30fea59733-plugins-config-volume" pod "06211214-57e4-432e-b254-0b30fea59733" (UID: "06211214-57e4-432e-b254-0b30fea59733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:56.549735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.549713 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rj7x\" (UniqueName: \"kubernetes.io/projected/7c94fbff-b58f-4434-a509-01f353a46658-kube-api-access-2rj7x\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:10:56.549832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.549736 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06211214-57e4-432e-b254-0b30fea59733-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:10:56.549832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.549748 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94fbff-b58f-4434-a509-01f353a46658-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:10:56.552081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.552058 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06211214-57e4-432e-b254-0b30fea59733-kube-api-access-j9cc2" pod "06211214-57e4-432e-b254-0b30fea59733" (UID: "06211214-57e4-432e-b254-0b30fea59733"). InnerVolumeSpecName "kube-api-access-j9cc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:56.651268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:56.651225 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9cc2\" (UniqueName: \"kubernetes.io/projected/06211214-57e4-432e-b254-0b30fea59733-kube-api-access-j9cc2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:10:57.217420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.217396 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" Oct 30 01:10:57.217868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.217420 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9" event={"ID":"06211214-57e4-432e-b254-0b30fea59733","Type":"ContainerDied","Data":"6bd257c9479f4cb00b977ed2dedcf27320f4e14fa3499832a45443c220ffff36"} Oct 30 01:10:57.217868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.217484 4905 scope.go:122] "RemoveContainer" containerID="30b65753688a50a8b29d5cd1b39999911ac0cd0d0dd6f4910d88b88bf492f99d" Oct 30 01:10:57.219318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.219114 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" Oct 30 01:10:57.219318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.219135 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25" event={"ID":"7c94fbff-b58f-4434-a509-01f353a46658","Type":"ContainerDied","Data":"204ef9699a6b5f15bb47883a6846c69090d42cc823bb88042fd10dab1a843263"} Oct 30 01:10:57.230793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.230765 4905 scope.go:122] "RemoveContainer" containerID="ad9362b620589eb61f4eea97305ee9cdaee5820e3f8ee1de626616ab1e0bb9a6" Oct 30 01:10:57.276339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.276301 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25"] Oct 30 01:10:57.288925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.288894 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ca20f-78d87f84b8-ssm25"] Oct 30 01:10:57.325369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.325337 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9"] Oct 30 01:10:57.332948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:57.332912 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ca20f-5bb8c6bf95-bxlp9"] Oct 30 01:10:59.197212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:59.197174 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06211214-57e4-432e-b254-0b30fea59733" path="/var/lib/kubelet/pods/06211214-57e4-432e-b254-0b30fea59733/volumes" Oct 30 01:10:59.198000 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:10:59.197979 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c94fbff-b58f-4434-a509-01f353a46658" path="/var/lib/kubelet/pods/7c94fbff-b58f-4434-a509-01f353a46658/volumes" Oct 30 01:11:02.344706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:02.344661 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8"] Oct 30 01:11:02.345251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:02.344905 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" podUID="89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" containerName="maas-api" containerID="cri-o://d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2" gracePeriod=30 Oct 30 01:11:02.902540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:02.902493 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:11:03.185098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.177698 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-metrics-certs\") pod \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " Oct 30 01:11:03.185098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.177942 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-openshift-service-ca\") pod \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " Oct 30 01:11:03.185098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.177986 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-maas-api-tls\") pod \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " Oct 30 01:11:03.185098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.178081 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-kube-api-access-p9tz2\" (UniqueName: \"kubernetes.io/projected/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-kube-api-access-p9tz2\") pod \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\" (UID: \"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e\") " Oct 30 01:11:03.185098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.179654 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-openshift-service-ca" pod "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" (UID: "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:11:03.268503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.268469 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-kube-api-access-p9tz2" pod "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" (UID: "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e"). InnerVolumeSpecName "kube-api-access-p9tz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:03.272495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.272461 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-maas-api-tls" pod "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" (UID: "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:11:03.285146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.285109 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-metrics-certs" pod "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" (UID: "89c9bf62-06dc-4e3c-9271-a23ed7b5c13e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:11:03.285770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.285744 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:03.285932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.285919 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:03.286039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.286025 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9tz2\" (UniqueName: \"kubernetes.io/projected/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-kube-api-access-p9tz2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:03.286131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.286120 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:03.343387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.343344 4905 generic.go:356] "Generic (PLEG): container finished" podID="89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" containerID="d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2" exitCode=0 Oct 30 01:11:03.343742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.343713 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" event={"ID":"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e","Type":"ContainerDied","Data":"d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2"} Oct 30 01:11:03.343858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.343837 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" event={"ID":"89c9bf62-06dc-4e3c-9271-a23ed7b5c13e","Type":"ContainerDied","Data":"2f498efcf9d3a2e50d30c744bfd22eae4689718daf93bad7c4e0ea48d0410ba7"} Oct 30 01:11:03.343922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.343892 4905 scope.go:122] "RemoveContainer" containerID="d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2" Oct 30 01:11:03.344270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.344122 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8" Oct 30 01:11:03.394676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.394474 4905 scope.go:122] "RemoveContainer" containerID="d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2" Oct 30 01:11:03.394949 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:11:03.394846 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2\": container with ID starting with d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2 not found: ID does not exist" containerID="d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2" Oct 30 01:11:03.394949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.394894 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2"} err="failed to get container status \"d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2\": rpc error: code = NotFound desc = could not find container \"d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2\": container with ID starting with d68dcba2a0c4e670603b5a0ebc816d7a5b80ce4ecb5b12240816a3761abe73d2 not found: ID does not exist" Oct 30 01:11:03.438088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.438003 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8"] Oct 30 01:11:03.446570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:03.446482 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ca20f-57959dbf7f-lxxj8"] Oct 30 01:11:05.197157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:05.197119 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" path="/var/lib/kubelet/pods/89c9bf62-06dc-4e3c-9271-a23ed7b5c13e/volumes" Oct 30 01:11:10.348591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:10.348553 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w"] Oct 30 01:11:10.349238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:10.348854 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" containerID="cri-o://2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363" gracePeriod=30 Oct 30 01:11:12.421110 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:11:12.421069 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:11:15.702621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.702595 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:11:15.796200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796151 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-envoy\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796262 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-data\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796370 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-certs\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796497 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-kube-api-access-c7rb8\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-kube-api-access-c7rb8\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796686 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796581 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-credential-socket\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796686 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796668 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-token\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-token\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796717 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istiod-ca-cert\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796762 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-data" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:15.796914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796803 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-socket\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.796914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.796899 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-podinfo\") pod \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\" (UID: \"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56\") " Oct 30 01:11:15.797506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.797224 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-credential-socket" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:15.797506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.797352 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istiod-ca-cert" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:11:15.797703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.797597 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-socket" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:15.797703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.797509 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-certs" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:15.799037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.799018 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.799186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.799172 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.799278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.799266 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.799357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.799347 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.799453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.799430 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.799453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.799408 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-envoy" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:15.800791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.800766 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-kube-api-access-c7rb8" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "kube-api-access-c7rb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:15.801740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.801719 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-token" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:15.817064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.817023 4905 generic.go:356] "Generic (PLEG): container finished" podID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerID="2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363" exitCode=0 Oct 30 01:11:15.817393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.817367 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" event={"ID":"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56","Type":"ContainerDied","Data":"2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363"} Oct 30 01:11:15.817510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.817433 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" event={"ID":"0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56","Type":"ContainerDied","Data":"ccb1b93f05a2b073ea2e82d945a95babb8dceaf242835d623ad1cff4d0ca5821"} Oct 30 01:11:15.817510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.817464 4905 scope.go:122] "RemoveContainer" containerID="2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363" Oct 30 01:11:15.817688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.817670 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" Oct 30 01:11:15.827082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.827050 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-podinfo" pod "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" (UID: "0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:11:15.848709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.848677 4905 scope.go:122] "RemoveContainer" containerID="2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1" Oct 30 01:11:15.860671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.860644 4905 scope.go:122] "RemoveContainer" containerID="2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363" Oct 30 01:11:15.861004 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:11:15.860977 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363\": container with ID starting with 2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363 not found: ID does not exist" containerID="2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363" Oct 30 01:11:15.861092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.861013 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363"} err="failed to get container status \"2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363\": rpc error: code = NotFound desc = could not find container \"2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363\": container with ID starting with 2e2a87838771df97aa8d2351e30136645a07a45703e3cb54384960c675268363 not found: ID does not exist" Oct 30 01:11:15.861092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.861035 4905 scope.go:122] "RemoveContainer" containerID="2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1" Oct 30 01:11:15.861321 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:11:15.861301 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1\": container with ID starting with 2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1 not found: ID does not exist" containerID="2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1" Oct 30 01:11:15.861381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.861330 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1"} err="failed to get container status \"2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1\": rpc error: code = NotFound desc = could not find container \"2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1\": container with ID starting with 2067c844ca9f108b1757820b9ad624f6d8225b4c6a0c13d329b3321fcea783d1 not found: ID does not exist" Oct 30 01:11:15.901050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.901013 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7rb8\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-kube-api-access-c7rb8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.901050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.901047 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.901050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.901059 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:15.901315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:15.901069 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:16.171559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.171510 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w"] Oct 30 01:11:16.183945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.183909 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w"] Oct 30 01:11:16.399409 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.399315 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh"] Oct 30 01:11:16.400743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400718 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" Oct 30 01:11:16.400743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400740 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400750 4905 cpu_manager.go:414] "containerMap: removing container" podUID="06211214-57e4-432e-b254-0b30fea59733" containerName="payload-pre-processing" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400755 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06211214-57e4-432e-b254-0b30fea59733" containerName="payload-pre-processing" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400778 4905 cpu_manager.go:414] "containerMap: removing container" podUID="89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" containerName="maas-api" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400783 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" containerName="maas-api" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400806 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400811 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400821 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7c94fbff-b58f-4434-a509-01f353a46658" containerName="payload-processing" Oct 30 01:11:16.400904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400826 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c94fbff-b58f-4434-a509-01f353a46658" containerName="payload-processing" Oct 30 01:11:16.401153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400943 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" Oct 30 01:11:16.401153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400952 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="06211214-57e4-432e-b254-0b30fea59733" containerName="payload-pre-processing" Oct 30 01:11:16.401153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400962 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c9bf62-06dc-4e3c-9271-a23ed7b5c13e" containerName="maas-api" Oct 30 01:11:16.401153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400972 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" Oct 30 01:11:16.401153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.400980 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c94fbff-b58f-4434-a509-01f353a46658" containerName="payload-processing" Oct 30 01:11:16.406749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.406723 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.432994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.430103 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh"] Oct 30 01:11:16.432994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.431197 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-95fe33-openshift-default-dockercfg-t66db\"" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.527861 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-credential-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.527969 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-envoy\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.528026 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtxz\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-kube-api-access-tdtxz\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.528100 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-certs\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.528138 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-token\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.528265 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.528413 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-podinfo\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.528465 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-data\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.529547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.528544 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istiod-ca-cert\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.621709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.621668 4905 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:11:16.622040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.622005 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-b7f5w" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:11:16.631911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.631869 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-envoy\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.632198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.632179 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtxz\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-kube-api-access-tdtxz\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.632370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.632353 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-certs\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.632539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.632503 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-token\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.632811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.632792 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.633139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.633121 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-podinfo\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.633305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.633287 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-data\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.633465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.633449 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istiod-ca-cert\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.650596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.641647 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-credential-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.650596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.638934 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-data\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.650596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.639157 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.650596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.639817 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istiod-ca-cert\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.650596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.638682 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-certs\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.650596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.642462 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-credential-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.666469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.666429 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-podinfo\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.666981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.666955 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-envoy\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.702540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.699759 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtxz\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-kube-api-access-tdtxz\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.751488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.750945 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-token\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:16.752010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:16.751760 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:17.238493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.236835 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56" path="/var/lib/kubelet/pods/0aeca3ce-019e-4b5a-9f23-6dbdc2aceb56/volumes" Oct 30 01:11:17.241024 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:11:17.240992 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4c4e40_abc2_4612_9b8e_fa2c3ba67485.slice/crio-817557aaed213972f3dbe4f06a46d1ae9cd90788c117c0bb0c5b9418e59cf1c8 WatchSource:0}: Error finding container 817557aaed213972f3dbe4f06a46d1ae9cd90788c117c0bb0c5b9418e59cf1c8: Status 404 returned error can't find the container with id 817557aaed213972f3dbe4f06a46d1ae9cd90788c117c0bb0c5b9418e59cf1c8 Oct 30 01:11:17.241315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.241291 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh"] Oct 30 01:11:17.244248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.244212 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:11:17.244336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.244287 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:11:17.244336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.244317 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:11:17.859284 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.859158 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" event={"ID":"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485","Type":"ContainerStarted","Data":"6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4"} Oct 30 01:11:17.859284 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.859240 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" event={"ID":"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485","Type":"ContainerStarted","Data":"817557aaed213972f3dbe4f06a46d1ae9cd90788c117c0bb0c5b9418e59cf1c8"} Oct 30 01:11:17.956337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:17.956282 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podStartSLOduration=1.956265774 podStartE2EDuration="1.956265774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:17.915716254 +0000 UTC m=+1962.858889447" watchObservedRunningTime="2026-10-30 01:11:17.956265774 +0000 UTC m=+1962.899438968" Oct 30 01:11:18.752991 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:18.752954 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:18.759730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:18.759702 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:18.869385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:18.869355 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:18.870664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:18.870632 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:11:21.332051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:21.332007 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps"] Oct 30 01:11:21.373865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:21.373832 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf"] Oct 30 01:11:22.980993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:22.980955 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv"] Oct 30 01:11:22.988351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:22.988323 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.010622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.010590 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-95fe33-dockercfg-jcplt\"" Oct 30 01:11:23.010807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.010681 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-95fe33\"" Oct 30 01:11:23.013939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.013682 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv"] Oct 30 01:11:23.390752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.390629 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4wh\" (UniqueName: \"kubernetes.io/projected/1b8c09b6-109b-4b35-9789-5ad9b705efe6-kube-api-access-9l4wh\") pod \"payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.393132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.393094 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c09b6-109b-4b35-9789-5ad9b705efe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.418509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.418474 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj"] Oct 30 01:11:23.440666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.438162 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.458149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.458118 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj"] Oct 30 01:11:23.514712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.514658 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5sh\" (UniqueName: \"kubernetes.io/projected/f2b68575-298b-487d-9112-6f387aefaa50-kube-api-access-vz5sh\") pod \"payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.514941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.514718 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c09b6-109b-4b35-9789-5ad9b705efe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.514941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.514838 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4wh\" (UniqueName: \"kubernetes.io/projected/1b8c09b6-109b-4b35-9789-5ad9b705efe6-kube-api-access-9l4wh\") pod \"payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.514941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.514898 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b68575-298b-487d-9112-6f387aefaa50-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.515563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.515539 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c09b6-109b-4b35-9789-5ad9b705efe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.535502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.535470 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4wh\" (UniqueName: \"kubernetes.io/projected/1b8c09b6-109b-4b35-9789-5ad9b705efe6-kube-api-access-9l4wh\") pod \"payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.610847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.610809 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:23.616990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.616946 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b68575-298b-487d-9112-6f387aefaa50-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.617203 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.617188 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5sh\" (UniqueName: \"kubernetes.io/projected/f2b68575-298b-487d-9112-6f387aefaa50-kube-api-access-vz5sh\") pod \"payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.617691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.617670 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b68575-298b-487d-9112-6f387aefaa50-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.639502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.639464 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5sh\" (UniqueName: \"kubernetes.io/projected/f2b68575-298b-487d-9112-6f387aefaa50-kube-api-access-vz5sh\") pod \"payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.760861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.760828 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:23.924619 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:11:23.924548 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8c09b6_109b_4b35_9789_5ad9b705efe6.slice/crio-fb7b1a67db06e75e294bafe832e3dc953127529690cae1aa0c9e86670783c8b5 WatchSource:0}: Error finding container fb7b1a67db06e75e294bafe832e3dc953127529690cae1aa0c9e86670783c8b5: Status 404 returned error can't find the container with id fb7b1a67db06e75e294bafe832e3dc953127529690cae1aa0c9e86670783c8b5 Oct 30 01:11:23.925300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:23.925266 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv"] Oct 30 01:11:24.076853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.076818 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj"] Oct 30 01:11:24.079215 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:11:24.079179 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b68575_298b_487d_9112_6f387aefaa50.slice/crio-43305f581e1a4032bc556afa5a2a1d97b42a9b46badefaaca8040968d3ad3c7c WatchSource:0}: Error finding container 43305f581e1a4032bc556afa5a2a1d97b42a9b46badefaaca8040968d3ad3c7c: Status 404 returned error can't find the container with id 43305f581e1a4032bc556afa5a2a1d97b42a9b46badefaaca8040968d3ad3c7c Oct 30 01:11:24.927668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.927623 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" event={"ID":"f2b68575-298b-487d-9112-6f387aefaa50","Type":"ContainerStarted","Data":"d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828"} Oct 30 01:11:24.927895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.927702 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" event={"ID":"f2b68575-298b-487d-9112-6f387aefaa50","Type":"ContainerStarted","Data":"43305f581e1a4032bc556afa5a2a1d97b42a9b46badefaaca8040968d3ad3c7c"} Oct 30 01:11:24.927895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.927772 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:24.930316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.930275 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" event={"ID":"1b8c09b6-109b-4b35-9789-5ad9b705efe6","Type":"ContainerStarted","Data":"90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43"} Oct 30 01:11:24.930463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.930342 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" event={"ID":"1b8c09b6-109b-4b35-9789-5ad9b705efe6","Type":"ContainerStarted","Data":"fb7b1a67db06e75e294bafe832e3dc953127529690cae1aa0c9e86670783c8b5"} Oct 30 01:11:24.930463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.930374 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:24.972880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:24.972825 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" podStartSLOduration=2.972809357 podStartE2EDuration="2.972809357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:24.972602781 +0000 UTC m=+1969.915775974" watchObservedRunningTime="2026-10-30 01:11:24.972809357 +0000 UTC m=+1969.915982549" Oct 30 01:11:25.014548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:25.014461 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" podStartSLOduration=3.014446656 podStartE2EDuration="3.014446656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:25.012851232 +0000 UTC m=+1969.956024425" watchObservedRunningTime="2026-10-30 01:11:25.014446656 +0000 UTC m=+1969.957619849" Oct 30 01:11:26.332945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.332884 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" podUID="f49a1306-296b-4c8a-99e9-8f1698a6399d" containerName="payload-processing" containerID="cri-o://53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce" gracePeriod=25 Oct 30 01:11:26.374319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.374271 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" podUID="38d0937f-af4c-40d0-add3-9361fa320b50" containerName="payload-pre-processing" containerID="cri-o://ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11" gracePeriod=25 Oct 30 01:11:26.643779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.643747 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:11:26.782147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.782107 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49a1306-296b-4c8a-99e9-8f1698a6399d-kube-api-access-j5459\" (UniqueName: \"kubernetes.io/projected/f49a1306-296b-4c8a-99e9-8f1698a6399d-kube-api-access-j5459\") pod \"f49a1306-296b-4c8a-99e9-8f1698a6399d\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " Oct 30 01:11:26.782321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.782195 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49a1306-296b-4c8a-99e9-8f1698a6399d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49a1306-296b-4c8a-99e9-8f1698a6399d-plugins-config-volume\") pod \"f49a1306-296b-4c8a-99e9-8f1698a6399d\" (UID: \"f49a1306-296b-4c8a-99e9-8f1698a6399d\") " Oct 30 01:11:26.782648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.782626 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49a1306-296b-4c8a-99e9-8f1698a6399d-plugins-config-volume" pod "f49a1306-296b-4c8a-99e9-8f1698a6399d" (UID: "f49a1306-296b-4c8a-99e9-8f1698a6399d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:11:26.783067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.783045 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49a1306-296b-4c8a-99e9-8f1698a6399d-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:26.784682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.784655 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49a1306-296b-4c8a-99e9-8f1698a6399d-kube-api-access-j5459" pod "f49a1306-296b-4c8a-99e9-8f1698a6399d" (UID: "f49a1306-296b-4c8a-99e9-8f1698a6399d"). InnerVolumeSpecName "kube-api-access-j5459". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:26.884585 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.884450 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5459\" (UniqueName: \"kubernetes.io/projected/f49a1306-296b-4c8a-99e9-8f1698a6399d-kube-api-access-j5459\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:26.958768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.958726 4905 generic.go:356] "Generic (PLEG): container finished" podID="f49a1306-296b-4c8a-99e9-8f1698a6399d" containerID="53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce" exitCode=0 Oct 30 01:11:26.959007 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.958978 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" Oct 30 01:11:26.959100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.959007 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" event={"ID":"f49a1306-296b-4c8a-99e9-8f1698a6399d","Type":"ContainerDied","Data":"53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce"} Oct 30 01:11:26.959100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.959047 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps" event={"ID":"f49a1306-296b-4c8a-99e9-8f1698a6399d","Type":"ContainerDied","Data":"f219fb209c05a223335eb98f21f69e5af36a6fb3f2f3e672f9572bf587cf8fee"} Oct 30 01:11:26.959100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.959069 4905 scope.go:122] "RemoveContainer" containerID="53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce" Oct 30 01:11:26.972134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.972107 4905 scope.go:122] "RemoveContainer" containerID="53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce" Oct 30 01:11:26.972502 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:11:26.972476 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce\": container with ID starting with 53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce not found: ID does not exist" containerID="53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce" Oct 30 01:11:26.972639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:26.972510 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce"} err="failed to get container status \"53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce\": rpc error: code = NotFound desc = could not find container \"53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce\": container with ID starting with 53758a8148c2aeb0d517d210a144f4db583a237cd5bafbf3a98aeaf542fab9ce not found: ID does not exist" Oct 30 01:11:27.024680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:27.024637 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps"] Oct 30 01:11:27.038173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:27.038133 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b483fd-69f7cf597b-jvtps"] Oct 30 01:11:27.194351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:27.194260 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49a1306-296b-4c8a-99e9-8f1698a6399d" path="/var/lib/kubelet/pods/f49a1306-296b-4c8a-99e9-8f1698a6399d/volumes" Oct 30 01:11:28.399730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:28.399689 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" podUID="38d0937f-af4c-40d0-add3-9361fa320b50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:11:35.944087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:35.944054 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:11:35.944564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:35.944503 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:11:38.399107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:38.399061 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" podUID="38d0937f-af4c-40d0-add3-9361fa320b50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:11:42.418391 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:11:42.418351 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:11:47.098138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.098109 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:11:47.202250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.202158 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d0937f-af4c-40d0-add3-9361fa320b50-kube-api-access-zzkhm\" (UniqueName: \"kubernetes.io/projected/38d0937f-af4c-40d0-add3-9361fa320b50-kube-api-access-zzkhm\") pod \"38d0937f-af4c-40d0-add3-9361fa320b50\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " Oct 30 01:11:47.202250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.202199 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d0937f-af4c-40d0-add3-9361fa320b50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d0937f-af4c-40d0-add3-9361fa320b50-plugins-config-volume\") pod \"38d0937f-af4c-40d0-add3-9361fa320b50\" (UID: \"38d0937f-af4c-40d0-add3-9361fa320b50\") " Oct 30 01:11:47.202762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.202733 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d0937f-af4c-40d0-add3-9361fa320b50-plugins-config-volume" pod "38d0937f-af4c-40d0-add3-9361fa320b50" (UID: "38d0937f-af4c-40d0-add3-9361fa320b50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:11:47.203389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.203368 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d0937f-af4c-40d0-add3-9361fa320b50-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:47.205497 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.205474 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d0937f-af4c-40d0-add3-9361fa320b50-kube-api-access-zzkhm" pod "38d0937f-af4c-40d0-add3-9361fa320b50" (UID: "38d0937f-af4c-40d0-add3-9361fa320b50"). InnerVolumeSpecName "kube-api-access-zzkhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:47.305238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.305199 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzkhm\" (UniqueName: \"kubernetes.io/projected/38d0937f-af4c-40d0-add3-9361fa320b50-kube-api-access-zzkhm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:11:47.386227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.386191 4905 generic.go:356] "Generic (PLEG): container finished" podID="38d0937f-af4c-40d0-add3-9361fa320b50" containerID="ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11" exitCode=0 Oct 30 01:11:47.386407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.386303 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" Oct 30 01:11:47.386407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.386323 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" event={"ID":"38d0937f-af4c-40d0-add3-9361fa320b50","Type":"ContainerDied","Data":"ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11"} Oct 30 01:11:47.386407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.386381 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf" event={"ID":"38d0937f-af4c-40d0-add3-9361fa320b50","Type":"ContainerDied","Data":"aa1fbc884bc89c94e83f864a4f9c9cbaebd846dc717e540f41221c38a02aabd4"} Oct 30 01:11:47.386407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.386404 4905 scope.go:122] "RemoveContainer" containerID="ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11" Oct 30 01:11:47.397970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.397951 4905 scope.go:122] "RemoveContainer" containerID="ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11" Oct 30 01:11:47.398244 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:11:47.398223 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11\": container with ID starting with ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11 not found: ID does not exist" containerID="ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11" Oct 30 01:11:47.398306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.398251 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11"} err="failed to get container status \"ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11\": rpc error: code = NotFound desc = could not find container \"ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11\": container with ID starting with ff7032ebb0d74a6144efaf63ada11f25b69cecc691c0eb768390ac61888b8c11 not found: ID does not exist" Oct 30 01:11:47.444380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.444355 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf"] Oct 30 01:11:47.449235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:47.449214 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b483fd-5d4f6d8fff-2lwhf"] Oct 30 01:11:49.217077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:49.217032 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d0937f-af4c-40d0-add3-9361fa320b50" path="/var/lib/kubelet/pods/38d0937f-af4c-40d0-add3-9361fa320b50/volumes" Oct 30 01:11:51.321576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:51.321535 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q"] Oct 30 01:11:51.388063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:51.388028 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d"] Oct 30 01:11:56.322673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:56.322628 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" podUID="b5a09346-42db-439c-bef3-96c870f851e1" containerName="payload-processing" containerID="cri-o://71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92" gracePeriod=25 Oct 30 01:11:56.388913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:11:56.388866 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" podUID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" containerName="payload-pre-processing" containerID="cri-o://2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17" gracePeriod=25 Oct 30 01:12:00.618552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:00.616760 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx"] Oct 30 01:12:00.618552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:00.617539 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" containerID="cri-o://569dcc0f61dbeab789dfe9a0fac2302641e41f5b2cc99da393a6ecd28ed5e5ef" gracePeriod=30 Oct 30 01:12:04.179642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:04.179596 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" podUID="b5a09346-42db-439c-bef3-96c870f851e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:04.180894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:04.180838 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" podUID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:06.615957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.615921 4905 generic.go:356] "Generic (PLEG): container finished" podID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerID="569dcc0f61dbeab789dfe9a0fac2302641e41f5b2cc99da393a6ecd28ed5e5ef" exitCode=0 Oct 30 01:12:06.616469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.616004 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" event={"ID":"5d0996ac-76a8-4320-ad1d-7cad3c396659","Type":"ContainerDied","Data":"569dcc0f61dbeab789dfe9a0fac2302641e41f5b2cc99da393a6ecd28ed5e5ef"} Oct 30 01:12:06.616469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.616092 4905 scope.go:122] "RemoveContainer" containerID="c5fb03af62818c99ff130317894a7288500bec8feebd059fd9f0c5862a9ce362" Oct 30 01:12:06.755471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.755293 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:12:06.849438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849381 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-kube-api-access-pw5nq\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-kube-api-access-pw5nq\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849448 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-certs\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849503 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-credential-socket\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849563 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-podinfo\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849664 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-data\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849765 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-token\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-token\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849823 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-socket\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849885 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-envoy\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.849959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.849925 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0996ac-76a8-4320-ad1d-7cad3c396659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0996ac-76a8-4320-ad1d-7cad3c396659-istiod-ca-cert\") pod \"5d0996ac-76a8-4320-ad1d-7cad3c396659\" (UID: \"5d0996ac-76a8-4320-ad1d-7cad3c396659\") " Oct 30 01:12:06.850770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.850733 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-certs" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:06.850770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.850748 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-socket" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:06.850965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.850950 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-data" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:06.851278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.851255 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-credential-socket" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:06.851428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.851339 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0996ac-76a8-4320-ad1d-7cad3c396659-istiod-ca-cert" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:06.852194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.852168 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.852291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.852198 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.852291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.852214 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.852291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.852230 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.852291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.852243 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0996ac-76a8-4320-ad1d-7cad3c396659-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.857149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.857103 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-envoy" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:06.857149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.857109 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-podinfo" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:06.857351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.857258 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-kube-api-access-pw5nq" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "kube-api-access-pw5nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:06.858976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.858946 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-token" pod "5d0996ac-76a8-4320-ad1d-7cad3c396659" (UID: "5d0996ac-76a8-4320-ad1d-7cad3c396659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:06.953956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.953914 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.953956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.953951 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.954237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.953965 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0996ac-76a8-4320-ad1d-7cad3c396659-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:06.954237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:06.953979 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw5nq\" (UniqueName: \"kubernetes.io/projected/5d0996ac-76a8-4320-ad1d-7cad3c396659-kube-api-access-pw5nq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:07.626382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:07.626344 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" Oct 30 01:12:07.626894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:07.626347 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx" event={"ID":"5d0996ac-76a8-4320-ad1d-7cad3c396659","Type":"ContainerDied","Data":"594cdce0792e8e5f925e29aec52cc199042ed244e86ffbf5e99777e82a15f248"} Oct 30 01:12:07.626894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:07.626507 4905 scope.go:122] "RemoveContainer" containerID="569dcc0f61dbeab789dfe9a0fac2302641e41f5b2cc99da393a6ecd28ed5e5ef" Oct 30 01:12:07.689848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:07.685303 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw"] Oct 30 01:12:07.689848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:07.685555 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" containerID="cri-o://5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31" gracePeriod=30 Oct 30 01:12:07.741124 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:07.741075 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx"] Oct 30 01:12:07.747879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:07.747845 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-nskcx"] Oct 30 01:12:09.194646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:09.194614 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" path="/var/lib/kubelet/pods/5d0996ac-76a8-4320-ad1d-7cad3c396659/volumes" Oct 30 01:12:11.963722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:11.963675 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g"] Oct 30 01:12:12.480589 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:12.480502 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:12:13.549961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.549885 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:12:13.603321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603281 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-credential-socket\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.603508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603409 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-socket\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.603508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603469 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-podinfo\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.603508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603486 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-data\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.603723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603553 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-certs\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.603723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603634 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-credential-socket" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.603825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603754 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istiod-ca-cert\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.603825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603811 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-envoy\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.603941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603822 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-socket" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.603941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603830 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-data" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.603941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603843 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-certs" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.603941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603866 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-kube-api-access-cvhvw\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-kube-api-access-cvhvw\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.604215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.603950 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-token\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-token\") pod \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\" (UID: \"c1dadcfa-aabe-4763-ac48-e8cd508e4916\") " Oct 30 01:12:13.604284 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.604210 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istiod-ca-cert" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:13.604736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.604714 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.604881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.604738 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.604881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.604751 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.604881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.604763 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.604881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.604774 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.606166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.606145 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-podinfo" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:13.607089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.607062 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-envoy" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.607195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.607098 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-token" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:13.607195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.607166 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-kube-api-access-cvhvw" pod "c1dadcfa-aabe-4763-ac48-e8cd508e4916" (UID: "c1dadcfa-aabe-4763-ac48-e8cd508e4916"). InnerVolumeSpecName "kube-api-access-cvhvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:13.706308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.706260 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.706308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.706307 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.706556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.706323 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvhvw\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-kube-api-access-cvhvw\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.706556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.706333 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dadcfa-aabe-4763-ac48-e8cd508e4916-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.746047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.746009 4905 generic.go:356] "Generic (PLEG): container finished" podID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerID="5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31" exitCode=0 Oct 30 01:12:13.746313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.746282 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" event={"ID":"c1dadcfa-aabe-4763-ac48-e8cd508e4916","Type":"ContainerDied","Data":"5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31"} Oct 30 01:12:13.746381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.746355 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" event={"ID":"c1dadcfa-aabe-4763-ac48-e8cd508e4916","Type":"ContainerDied","Data":"e44569ec89574b413b3d255cdee59a9e47144ed10894a2b3ba930806b30c51a2"} Oct 30 01:12:13.746420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.746382 4905 scope.go:122] "RemoveContainer" containerID="5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31" Oct 30 01:12:13.746570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.746556 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw" Oct 30 01:12:13.763748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.763726 4905 scope.go:122] "RemoveContainer" containerID="5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467" Oct 30 01:12:13.775974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.775948 4905 scope.go:122] "RemoveContainer" containerID="5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31" Oct 30 01:12:13.776304 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:13.776279 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31\": container with ID starting with 5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31 not found: ID does not exist" containerID="5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31" Oct 30 01:12:13.776405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.776314 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31"} err="failed to get container status \"5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31\": rpc error: code = NotFound desc = could not find container \"5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31\": container with ID starting with 5a535f20f2bf20094cd8b453be2e41241b8a5503bc5d83702858323c6df5de31 not found: ID does not exist" Oct 30 01:12:13.776405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.776334 4905 scope.go:122] "RemoveContainer" containerID="5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467" Oct 30 01:12:13.776796 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:13.776774 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467\": container with ID starting with 5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467 not found: ID does not exist" containerID="5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467" Oct 30 01:12:13.776864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.776804 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467"} err="failed to get container status \"5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467\": rpc error: code = NotFound desc = could not find container \"5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467\": container with ID starting with 5572e13f792980b33c45714d0e2a6cb994770616a65e233ebb31c1123ab1c467 not found: ID does not exist" Oct 30 01:12:13.810341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.810244 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw"] Oct 30 01:12:13.814703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:13.814674 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-ssrrw"] Oct 30 01:12:14.246280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:14.245265 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" podUID="b5a09346-42db-439c-bef3-96c870f851e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:14.265596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:14.261668 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" podUID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:15.194795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:15.194652 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" path="/var/lib/kubelet/pods/c1dadcfa-aabe-4763-ac48-e8cd508e4916/volumes" Oct 30 01:12:16.757741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.757708 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:12:16.761905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.761877 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:12:16.933909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.933809 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-plugins-config-volume\") pod \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " Oct 30 01:12:16.934099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.933912 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-kube-api-access-lnbhz\" (UniqueName: \"kubernetes.io/projected/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-kube-api-access-lnbhz\") pod \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\" (UID: \"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef\") " Oct 30 01:12:16.934099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.934041 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a09346-42db-439c-bef3-96c870f851e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a09346-42db-439c-bef3-96c870f851e1-plugins-config-volume\") pod \"b5a09346-42db-439c-bef3-96c870f851e1\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " Oct 30 01:12:16.934227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.934169 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-plugins-config-volume" pod "da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" (UID: "da4510b7-bf39-4a09-9cf2-ad5c1ac93aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:16.934336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.934306 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a09346-42db-439c-bef3-96c870f851e1-kube-api-access-4vx72\" (UniqueName: \"kubernetes.io/projected/b5a09346-42db-439c-bef3-96c870f851e1-kube-api-access-4vx72\") pod \"b5a09346-42db-439c-bef3-96c870f851e1\" (UID: \"b5a09346-42db-439c-bef3-96c870f851e1\") " Oct 30 01:12:16.934482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.934455 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a09346-42db-439c-bef3-96c870f851e1-plugins-config-volume" pod "b5a09346-42db-439c-bef3-96c870f851e1" (UID: "b5a09346-42db-439c-bef3-96c870f851e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:16.935027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.935005 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:16.935139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.935032 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a09346-42db-439c-bef3-96c870f851e1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:16.936488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.936466 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-kube-api-access-lnbhz" pod "da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" (UID: "da4510b7-bf39-4a09-9cf2-ad5c1ac93aef"). InnerVolumeSpecName "kube-api-access-lnbhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:16.936605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.936498 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a09346-42db-439c-bef3-96c870f851e1-kube-api-access-4vx72" pod "b5a09346-42db-439c-bef3-96c870f851e1" (UID: "b5a09346-42db-439c-bef3-96c870f851e1"). InnerVolumeSpecName "kube-api-access-4vx72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:16.947239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.947210 4905 generic.go:356] "Generic (PLEG): container finished" podID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" containerID="2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17" exitCode=0 Oct 30 01:12:16.947347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.947264 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" Oct 30 01:12:16.947347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.947289 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" event={"ID":"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef","Type":"ContainerDied","Data":"2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17"} Oct 30 01:12:16.947347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.947340 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d" event={"ID":"da4510b7-bf39-4a09-9cf2-ad5c1ac93aef","Type":"ContainerDied","Data":"9a5a6b887cd1d03b6de399c04ad8de43cea3359aaf9757f81daaa829cacbe394"} Oct 30 01:12:16.947482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.947363 4905 scope.go:122] "RemoveContainer" containerID="2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17" Oct 30 01:12:16.949304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.949265 4905 generic.go:356] "Generic (PLEG): container finished" podID="b5a09346-42db-439c-bef3-96c870f851e1" containerID="71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92" exitCode=0 Oct 30 01:12:16.949432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.949319 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" Oct 30 01:12:16.949432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.949320 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" event={"ID":"b5a09346-42db-439c-bef3-96c870f851e1","Type":"ContainerDied","Data":"71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92"} Oct 30 01:12:16.949432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.949373 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q" event={"ID":"b5a09346-42db-439c-bef3-96c870f851e1","Type":"ContainerDied","Data":"bfe464be243fa58681486586a897ce1fe6288a6198e9948a0ac46af3328d811e"} Oct 30 01:12:16.961583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.961564 4905 scope.go:122] "RemoveContainer" containerID="2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17" Oct 30 01:12:16.961935 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:16.961911 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17\": container with ID starting with 2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17 not found: ID does not exist" containerID="2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17" Oct 30 01:12:16.962033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.961944 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17"} err="failed to get container status \"2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17\": rpc error: code = NotFound desc = could not find container \"2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17\": container with ID starting with 2617b464a6bfc9e6d6b60a989fcd30211797d66327a4d7c851fd171a25bdbf17 not found: ID does not exist" Oct 30 01:12:16.962033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.961969 4905 scope.go:122] "RemoveContainer" containerID="71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92" Oct 30 01:12:16.964935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.964884 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" podUID="4139be30-705e-403f-944b-8013a773f730" containerName="payload-processing" containerID="cri-o://e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8" gracePeriod=25 Oct 30 01:12:16.975388 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.975369 4905 scope.go:122] "RemoveContainer" containerID="71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92" Oct 30 01:12:16.975760 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:16.975733 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92\": container with ID starting with 71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92 not found: ID does not exist" containerID="71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92" Oct 30 01:12:16.975910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:16.975771 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92"} err="failed to get container status \"71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92\": rpc error: code = NotFound desc = could not find container \"71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92\": container with ID starting with 71c2ade548f694e20d07abd88eeb16337cc018a61a8106eb8f6163e8d0a9ff92 not found: ID does not exist" Oct 30 01:12:17.006602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.006570 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d"] Oct 30 01:12:17.011025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.010997 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d9f1-56cd5c6bcf-fn77d"] Oct 30 01:12:17.036774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.036738 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vx72\" (UniqueName: \"kubernetes.io/projected/b5a09346-42db-439c-bef3-96c870f851e1-kube-api-access-4vx72\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:17.036774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.036766 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnbhz\" (UniqueName: \"kubernetes.io/projected/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef-kube-api-access-lnbhz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:17.043806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.043771 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q"] Oct 30 01:12:17.048167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.048140 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d9f1-6b554dd95d-j7w6q"] Oct 30 01:12:17.202068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.201985 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a09346-42db-439c-bef3-96c870f851e1" path="/var/lib/kubelet/pods/b5a09346-42db-439c-bef3-96c870f851e1/volumes" Oct 30 01:12:17.202762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.202738 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" path="/var/lib/kubelet/pods/da4510b7-bf39-4a09-9cf2-ad5c1ac93aef/volumes" Oct 30 01:12:17.257467 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.257434 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:12:17.444986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.444823 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4139be30-705e-403f-944b-8013a773f730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4139be30-705e-403f-944b-8013a773f730-plugins-config-volume\") pod \"4139be30-705e-403f-944b-8013a773f730\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " Oct 30 01:12:17.444986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.444943 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4139be30-705e-403f-944b-8013a773f730-kube-api-access-fbrxk\" (UniqueName: \"kubernetes.io/projected/4139be30-705e-403f-944b-8013a773f730-kube-api-access-fbrxk\") pod \"4139be30-705e-403f-944b-8013a773f730\" (UID: \"4139be30-705e-403f-944b-8013a773f730\") " Oct 30 01:12:17.445968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.445939 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4139be30-705e-403f-944b-8013a773f730-plugins-config-volume" pod "4139be30-705e-403f-944b-8013a773f730" (UID: "4139be30-705e-403f-944b-8013a773f730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:17.447897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.447870 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4139be30-705e-403f-944b-8013a773f730-kube-api-access-fbrxk" pod "4139be30-705e-403f-944b-8013a773f730" (UID: "4139be30-705e-403f-944b-8013a773f730"). InnerVolumeSpecName "kube-api-access-fbrxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:17.547093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.547047 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4139be30-705e-403f-944b-8013a773f730-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:17.547093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.547085 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbrxk\" (UniqueName: \"kubernetes.io/projected/4139be30-705e-403f-944b-8013a773f730-kube-api-access-fbrxk\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:17.972009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.971019 4905 generic.go:356] "Generic (PLEG): container finished" podID="4139be30-705e-403f-944b-8013a773f730" containerID="e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8" exitCode=0 Oct 30 01:12:17.972009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.971290 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" event={"ID":"4139be30-705e-403f-944b-8013a773f730","Type":"ContainerDied","Data":"e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8"} Oct 30 01:12:17.972009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.971387 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" event={"ID":"4139be30-705e-403f-944b-8013a773f730","Type":"ContainerDied","Data":"d94f2c94d06ee25dbb971a9c78103f705fcdee0bfdc3d20d51b1afc8eaa9f0cf"} Oct 30 01:12:17.972009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.971430 4905 scope.go:122] "RemoveContainer" containerID="e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8" Oct 30 01:12:17.972009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.971645 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g" Oct 30 01:12:17.991703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.991671 4905 scope.go:122] "RemoveContainer" containerID="e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8" Oct 30 01:12:17.992031 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:17.992007 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8\": container with ID starting with e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8 not found: ID does not exist" containerID="e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8" Oct 30 01:12:17.992115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:17.992040 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8"} err="failed to get container status \"e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8\": rpc error: code = NotFound desc = could not find container \"e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8\": container with ID starting with e149a246ed77dd142adf7c9691583bbdfa3116c31d79bad2499f01265f0764a8 not found: ID does not exist" Oct 30 01:12:18.030371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:18.030342 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g"] Oct 30 01:12:18.043780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:18.043749 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1eae0b-85c4b6b8b6-v6l6g"] Oct 30 01:12:19.193323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:19.193284 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4139be30-705e-403f-944b-8013a773f730" path="/var/lib/kubelet/pods/4139be30-705e-403f-944b-8013a773f730/volumes" Oct 30 01:12:22.578878 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:22.578838 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv"] Oct 30 01:12:22.616131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:22.616092 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv"] Oct 30 01:12:27.579366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:27.579323 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" podUID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" containerName="payload-processing" containerID="cri-o://d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db" gracePeriod=25 Oct 30 01:12:27.617381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:27.617318 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerName="payload-pre-processing" containerID="cri-o://44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6" gracePeriod=25 Oct 30 01:12:27.939346 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:27.939232 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:31.922719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:31.922681 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr"] Oct 30 01:12:31.923270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:31.922921 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" containerID="cri-o://082f7eda2fe1948bded6d130ac1da1569927b677648d5af28418cfaac57abe8c" gracePeriod=30 Oct 30 01:12:34.789111 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:34.789073 4905 patch_prober.go:29] interesting pod/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:12:34.789574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:34.789157 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:12:36.927693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:36.927652 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" podUID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:37.276823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.276779 4905 generic.go:356] "Generic (PLEG): container finished" podID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerID="082f7eda2fe1948bded6d130ac1da1569927b677648d5af28418cfaac57abe8c" exitCode=0 Oct 30 01:12:37.277144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.276839 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" event={"ID":"54ff57b2-7fb3-40bb-a720-7c35964885f2","Type":"ContainerDied","Data":"082f7eda2fe1948bded6d130ac1da1569927b677648d5af28418cfaac57abe8c"} Oct 30 01:12:37.277304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.277195 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" event={"ID":"54ff57b2-7fb3-40bb-a720-7c35964885f2","Type":"ContainerDied","Data":"2bcc00bd1fe7984b9edbeeba75d420e18ccd0f0c856f7a95f9ba516085d960f4"} Oct 30 01:12:37.277304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.277230 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bcc00bd1fe7984b9edbeeba75d420e18ccd0f0c856f7a95f9ba516085d960f4" Oct 30 01:12:37.277304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.277253 4905 scope.go:122] "RemoveContainer" containerID="e4d04365958fa41c2c086106555bc822b2e86567a3850358a525a1dd469cc708" Oct 30 01:12:37.292431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.292397 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:12:37.406415 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406378 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-podinfo\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.406641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406490 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-envoy\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.406719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406704 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-token\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-token\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.406780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406770 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-data\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.406838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406802 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-kube-api-access-zrkk7\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-kube-api-access-zrkk7\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.406907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406863 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-certs\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.406907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406892 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-credential-socket\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.407013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.406981 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-socket\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.407107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.407092 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ff57b2-7fb3-40bb-a720-7c35964885f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ff57b2-7fb3-40bb-a720-7c35964885f2-istiod-ca-cert\") pod \"54ff57b2-7fb3-40bb-a720-7c35964885f2\" (UID: \"54ff57b2-7fb3-40bb-a720-7c35964885f2\") " Oct 30 01:12:37.407260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.407232 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-credential-socket" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:37.407464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.407441 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-socket" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:37.407792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.407759 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ff57b2-7fb3-40bb-a720-7c35964885f2-istiod-ca-cert" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:37.408036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.408011 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.408155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.408044 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.408155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.408061 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ff57b2-7fb3-40bb-a720-7c35964885f2-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.408779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.408753 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-data" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:37.408974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.408942 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-certs" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:37.411084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.409985 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-envoy" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:37.411084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.410402 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-podinfo" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:37.412490 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.412461 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-token" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:37.412862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.412829 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-kube-api-access-zrkk7" pod "54ff57b2-7fb3-40bb-a720-7c35964885f2" (UID: "54ff57b2-7fb3-40bb-a720-7c35964885f2"). InnerVolumeSpecName "kube-api-access-zrkk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:37.509241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.509200 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.509241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.509236 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.509241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.509246 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrkk7\" (UniqueName: \"kubernetes.io/projected/54ff57b2-7fb3-40bb-a720-7c35964885f2-kube-api-access-zrkk7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.509595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.509257 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.509595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.509266 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.509595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.509276 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ff57b2-7fb3-40bb-a720-7c35964885f2-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.953109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:37.952959 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:38.290088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:38.290058 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr" Oct 30 01:12:38.392904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:38.392845 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr"] Oct 30 01:12:38.397638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:38.397604 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8e8ad1-openshift-default-5d7d9869bf-kd8rr"] Oct 30 01:12:39.196196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:39.196158 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" path="/var/lib/kubelet/pods/54ff57b2-7fb3-40bb-a720-7c35964885f2/volumes" Oct 30 01:12:42.418975 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:42.418936 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:12:42.860925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:42.860888 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t"] Oct 30 01:12:46.927478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:46.927433 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" podUID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:47.711233 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:47.711198 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692c5ba6_0bf9_4f73_909f_28a5fc55a572.slice/crio-44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692c5ba6_0bf9_4f73_909f_28a5fc55a572.slice/crio-conmon-44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:12:47.866940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:47.866592 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" podUID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" containerName="payload-processing" containerID="cri-o://910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7" gracePeriod=25 Oct 30 01:12:47.953543 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:12:47.949373 4905 logging.go:55] [core] [Channel #6003 SubChannel #6004]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.225:50052", ServerName: "10.132.0.225:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.225:50052: connect: connection refused" Oct 30 01:12:48.169014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.168691 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:12:48.221147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.221108 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ed45f5-c974-4e73-a92c-60189fe4dc03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ed45f5-c974-4e73-a92c-60189fe4dc03-plugins-config-volume\") pod \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " Oct 30 01:12:48.221325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.221220 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ed45f5-c974-4e73-a92c-60189fe4dc03-kube-api-access-7l94w\" (UniqueName: \"kubernetes.io/projected/e6ed45f5-c974-4e73-a92c-60189fe4dc03-kube-api-access-7l94w\") pod \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\" (UID: \"e6ed45f5-c974-4e73-a92c-60189fe4dc03\") " Oct 30 01:12:48.221555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.221511 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ed45f5-c974-4e73-a92c-60189fe4dc03-plugins-config-volume" pod "e6ed45f5-c974-4e73-a92c-60189fe4dc03" (UID: "e6ed45f5-c974-4e73-a92c-60189fe4dc03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:48.222174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.222154 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ed45f5-c974-4e73-a92c-60189fe4dc03-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:48.223821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.223795 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ed45f5-c974-4e73-a92c-60189fe4dc03-kube-api-access-7l94w" pod "e6ed45f5-c974-4e73-a92c-60189fe4dc03" (UID: "e6ed45f5-c974-4e73-a92c-60189fe4dc03"). InnerVolumeSpecName "kube-api-access-7l94w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:48.252043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.252015 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:12:48.322878 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.322843 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692c5ba6-0bf9-4f73-909f-28a5fc55a572-kube-api-access-wjvnh\" (UniqueName: \"kubernetes.io/projected/692c5ba6-0bf9-4f73-909f-28a5fc55a572-kube-api-access-wjvnh\") pod \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " Oct 30 01:12:48.323207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.323190 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692c5ba6-0bf9-4f73-909f-28a5fc55a572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692c5ba6-0bf9-4f73-909f-28a5fc55a572-plugins-config-volume\") pod \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\" (UID: \"692c5ba6-0bf9-4f73-909f-28a5fc55a572\") " Oct 30 01:12:48.323811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.323780 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692c5ba6-0bf9-4f73-909f-28a5fc55a572-plugins-config-volume" pod "692c5ba6-0bf9-4f73-909f-28a5fc55a572" (UID: "692c5ba6-0bf9-4f73-909f-28a5fc55a572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:48.323811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.323799 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l94w\" (UniqueName: \"kubernetes.io/projected/e6ed45f5-c974-4e73-a92c-60189fe4dc03-kube-api-access-7l94w\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:48.325587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.325556 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692c5ba6-0bf9-4f73-909f-28a5fc55a572-kube-api-access-wjvnh" pod "692c5ba6-0bf9-4f73-909f-28a5fc55a572" (UID: "692c5ba6-0bf9-4f73-909f-28a5fc55a572"). InnerVolumeSpecName "kube-api-access-wjvnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:48.418663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.415088 4905 generic.go:356] "Generic (PLEG): container finished" podID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerID="44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6" exitCode=0 Oct 30 01:12:48.418663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.415241 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" event={"ID":"692c5ba6-0bf9-4f73-909f-28a5fc55a572","Type":"ContainerDied","Data":"44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6"} Oct 30 01:12:48.418663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.415298 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" event={"ID":"692c5ba6-0bf9-4f73-909f-28a5fc55a572","Type":"ContainerDied","Data":"71042e567187423771cb78e32adda6374f19ca83392f7fa265f8c2da9e4c613e"} Oct 30 01:12:48.418663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.415335 4905 scope.go:122] "RemoveContainer" containerID="44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6" Oct 30 01:12:48.418663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.415601 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" Oct 30 01:12:48.462625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.458826 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692c5ba6-0bf9-4f73-909f-28a5fc55a572-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:48.462625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.458907 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvnh\" (UniqueName: \"kubernetes.io/projected/692c5ba6-0bf9-4f73-909f-28a5fc55a572-kube-api-access-wjvnh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:48.476466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.476429 4905 generic.go:356] "Generic (PLEG): container finished" podID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" containerID="d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db" exitCode=0 Oct 30 01:12:48.476713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.476691 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" event={"ID":"e6ed45f5-c974-4e73-a92c-60189fe4dc03","Type":"ContainerDied","Data":"d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db"} Oct 30 01:12:48.476809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.476793 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" event={"ID":"e6ed45f5-c974-4e73-a92c-60189fe4dc03","Type":"ContainerDied","Data":"e6dea4b9ba288cf910b829633c3ad2b393ad6ba5bab6f3c957723daa2917c355"} Oct 30 01:12:48.478224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.478196 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv" Oct 30 01:12:48.487940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.487918 4905 scope.go:122] "RemoveContainer" containerID="44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6" Oct 30 01:12:48.489019 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:48.488401 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6\": container with ID starting with 44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6 not found: ID does not exist" containerID="44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6" Oct 30 01:12:48.489019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.488430 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6"} err="failed to get container status \"44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6\": rpc error: code = NotFound desc = could not find container \"44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6\": container with ID starting with 44ee4042f32f805621bda6eb14e16dce6ffcd785448d12ebe36f8236f5599bf6 not found: ID does not exist" Oct 30 01:12:48.489019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.488448 4905 scope.go:122] "RemoveContainer" containerID="d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db" Oct 30 01:12:48.501054 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.501034 4905 scope.go:122] "RemoveContainer" containerID="d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db" Oct 30 01:12:48.501401 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:12:48.501376 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db\": container with ID starting with d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db not found: ID does not exist" containerID="d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db" Oct 30 01:12:48.501459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.501413 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db"} err="failed to get container status \"d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db\": rpc error: code = NotFound desc = could not find container \"d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db\": container with ID starting with d5754b601b636acdc51e808da5f3a0f0e02499df78cb78e33bfa74c74d31e4db not found: ID does not exist" Oct 30 01:12:48.538343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.538309 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv"] Oct 30 01:12:48.542924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.542896 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv"] Oct 30 01:12:48.579994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.579942 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv"] Oct 30 01:12:48.584079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.584055 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e8ad1-7c8b8894f4-7nqtv"] Oct 30 01:12:48.938592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:48.938535 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e8ad1-8659c645dc-ndfgv" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.225:50052\" within 1s: context deadline exceeded" Oct 30 01:12:49.193213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:49.193126 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" path="/var/lib/kubelet/pods/692c5ba6-0bf9-4f73-909f-28a5fc55a572/volumes" Oct 30 01:12:49.193716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:49.193639 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" path="/var/lib/kubelet/pods/e6ed45f5-c974-4e73-a92c-60189fe4dc03/volumes" Oct 30 01:12:50.320545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:50.318828 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7"] Oct 30 01:12:50.411106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:50.409683 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp"] Oct 30 01:12:51.227713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:51.227666 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" podUID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:12:55.319886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.319841 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" podUID="ed618312-e21b-412e-87f1-484cad8a0974" containerName="payload-processing" containerID="cri-o://ae378a280d1a871b8d3feb42e75df64036d95a7c626c71856e6e94e467d5154a" gracePeriod=25 Oct 30 01:12:55.410963 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.410923 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" podUID="9bc3397b-4c60-40f4-b0d0-950829147631" containerName="payload-pre-processing" containerID="cri-o://876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b" gracePeriod=25 Oct 30 01:12:55.603492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.603460 4905 generic.go:356] "Generic (PLEG): container finished" podID="ed618312-e21b-412e-87f1-484cad8a0974" containerID="ae378a280d1a871b8d3feb42e75df64036d95a7c626c71856e6e94e467d5154a" exitCode=0 Oct 30 01:12:55.603671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.603541 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" event={"ID":"ed618312-e21b-412e-87f1-484cad8a0974","Type":"ContainerDied","Data":"ae378a280d1a871b8d3feb42e75df64036d95a7c626c71856e6e94e467d5154a"} Oct 30 01:12:55.635245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.635216 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:12:55.743119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.743090 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed618312-e21b-412e-87f1-484cad8a0974-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed618312-e21b-412e-87f1-484cad8a0974-plugins-config-volume\") pod \"ed618312-e21b-412e-87f1-484cad8a0974\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " Oct 30 01:12:55.743327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.743237 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed618312-e21b-412e-87f1-484cad8a0974-kube-api-access-nv7zh\" (UniqueName: \"kubernetes.io/projected/ed618312-e21b-412e-87f1-484cad8a0974-kube-api-access-nv7zh\") pod \"ed618312-e21b-412e-87f1-484cad8a0974\" (UID: \"ed618312-e21b-412e-87f1-484cad8a0974\") " Oct 30 01:12:55.743861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.743828 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed618312-e21b-412e-87f1-484cad8a0974-plugins-config-volume" pod "ed618312-e21b-412e-87f1-484cad8a0974" (UID: "ed618312-e21b-412e-87f1-484cad8a0974"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:55.746406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.746383 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed618312-e21b-412e-87f1-484cad8a0974-kube-api-access-nv7zh" pod "ed618312-e21b-412e-87f1-484cad8a0974" (UID: "ed618312-e21b-412e-87f1-484cad8a0974"). InnerVolumeSpecName "kube-api-access-nv7zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:55.845374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.845345 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed618312-e21b-412e-87f1-484cad8a0974-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:55.845374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:55.845373 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv7zh\" (UniqueName: \"kubernetes.io/projected/ed618312-e21b-412e-87f1-484cad8a0974-kube-api-access-nv7zh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:12:56.612473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:56.612443 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" Oct 30 01:12:56.612939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:56.612472 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7" event={"ID":"ed618312-e21b-412e-87f1-484cad8a0974","Type":"ContainerDied","Data":"96ea3ec82a09df8b06dbe87367ca8802b08b5bd20fe8c774fce6ffb57182b072"} Oct 30 01:12:56.612939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:56.612554 4905 scope.go:122] "RemoveContainer" containerID="ae378a280d1a871b8d3feb42e75df64036d95a7c626c71856e6e94e467d5154a" Oct 30 01:12:56.672808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:56.672769 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7"] Oct 30 01:12:56.683867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:56.683835 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0680fa-59b9fdd958-dbjf7"] Oct 30 01:12:57.192584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:57.192511 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed618312-e21b-412e-87f1-484cad8a0974" path="/var/lib/kubelet/pods/ed618312-e21b-412e-87f1-484cad8a0974/volumes" Oct 30 01:12:58.763261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.763220 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb"] Oct 30 01:12:58.764802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764770 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" containerName="payload-processing" Oct 30 01:12:58.764802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764801 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764835 4905 cpu_manager.go:414] "containerMap: removing container" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764843 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764860 4905 cpu_manager.go:414] "containerMap: removing container" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerName="payload-pre-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764868 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerName="payload-pre-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764880 4905 cpu_manager.go:414] "containerMap: removing container" podUID="38d0937f-af4c-40d0-add3-9361fa320b50" containerName="payload-pre-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764888 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d0937f-af4c-40d0-add3-9361fa320b50" containerName="payload-pre-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764898 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f49a1306-296b-4c8a-99e9-8f1698a6399d" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764905 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49a1306-296b-4c8a-99e9-8f1698a6399d" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764923 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764931 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764948 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4139be30-705e-403f-944b-8013a773f730" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764955 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4139be30-705e-403f-944b-8013a773f730" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764970 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ed618312-e21b-412e-87f1-484cad8a0974" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.764978 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed618312-e21b-412e-87f1-484cad8a0974" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765000 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b5a09346-42db-439c-bef3-96c870f851e1" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765007 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a09346-42db-439c-bef3-96c870f851e1" containerName="payload-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765021 4905 cpu_manager.go:414] "containerMap: removing container" podUID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" containerName="payload-pre-processing" Oct 30 01:12:58.765032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765028 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" containerName="payload-pre-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765046 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765053 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765083 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765090 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765120 4905 cpu_manager.go:414] "containerMap: removing container" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765127 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765141 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765148 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765327 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765342 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765351 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765363 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ed45f5-c974-4e73-a92c-60189fe4dc03" containerName="payload-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765376 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ff57b2-7fb3-40bb-a720-7c35964885f2" containerName="istio-proxy" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765385 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="692c5ba6-0bf9-4f73-909f-28a5fc55a572" containerName="payload-pre-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765396 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d0937f-af4c-40d0-add3-9361fa320b50" containerName="payload-pre-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765405 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49a1306-296b-4c8a-99e9-8f1698a6399d" containerName="payload-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765417 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4139be30-705e-403f-944b-8013a773f730" containerName="payload-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765428 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed618312-e21b-412e-87f1-484cad8a0974" containerName="payload-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765436 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a09346-42db-439c-bef3-96c870f851e1" containerName="payload-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765447 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4510b7-bf39-4a09-9cf2-ad5c1ac93aef" containerName="payload-pre-processing" Oct 30 01:12:58.766026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.765457 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dadcfa-aabe-4763-ac48-e8cd508e4916" containerName="istio-proxy" Oct 30 01:12:58.770400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.770378 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.799452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.799418 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb"] Oct 30 01:12:58.806680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.806657 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-24d5ce-openshift-default-dockercfg-bwdz5\"" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912222 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-podinfo\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912309 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sz2m\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-kube-api-access-7sz2m\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912379 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-envoy\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912568 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-certs\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912618 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-credential-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912668 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-data\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912695 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912765 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istiod-ca-cert\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:58.927552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:58.912816 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-token\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.014908 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-podinfo\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.014978 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sz2m\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-kube-api-access-7sz2m\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.015040 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-envoy\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.015198 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-certs\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.015243 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-credential-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.015282 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-data\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.015307 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.015371 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istiod-ca-cert\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.015416 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-token\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.017006 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-credential-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.018347 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istiod-ca-cert\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.018741 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-data\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.019196 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.026648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.019241 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-certs\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.039216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.039188 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-podinfo\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.078671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.077839 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-envoy\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.081166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.081104 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-token\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.082103 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.082037 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sz2m\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-kube-api-access-7sz2m\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.099816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.099551 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:12:59.585443 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:12:59.585414 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6733ee_c5e2_4d99_8c1b_4d2a7077b76a.slice/crio-c3b0165037b2fa8859811a5408601f7fd27782ae3aa71a9edec07cd4e1f27b58 WatchSource:0}: Error finding container c3b0165037b2fa8859811a5408601f7fd27782ae3aa71a9edec07cd4e1f27b58: Status 404 returned error can't find the container with id c3b0165037b2fa8859811a5408601f7fd27782ae3aa71a9edec07cd4e1f27b58 Oct 30 01:12:59.587302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.587281 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb"] Oct 30 01:12:59.588294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.588259 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:12:59.588379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.588325 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:12:59.588379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.588357 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:12:59.654441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:12:59.654408 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" event={"ID":"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a","Type":"ContainerStarted","Data":"c3b0165037b2fa8859811a5408601f7fd27782ae3aa71a9edec07cd4e1f27b58"} Oct 30 01:13:00.665047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:00.665003 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" event={"ID":"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a","Type":"ContainerStarted","Data":"04486989f9ddf11a9951f190939bd89dc151ad9433ccecfeb3f313941f4191ef"} Oct 30 01:13:00.728582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:00.728506 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podStartSLOduration=2.728489137 podStartE2EDuration="2.728489137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:00.721440427 +0000 UTC m=+2065.664613625" watchObservedRunningTime="2026-10-30 01:13:00.728489137 +0000 UTC m=+2065.671662330" Oct 30 01:13:00.981890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:00.981850 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" podUID="9bc3397b-4c60-40f4-b0d0-950829147631" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:13:01.110547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:01.100675 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:13:01.124547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:01.117878 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:13:01.244593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:01.241625 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" podUID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:13:01.704641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:01.704548 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:13:01.712931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:01.712891 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:13:04.881096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:04.881055 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7"] Oct 30 01:13:04.881589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:04.881270 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" containerID="cri-o://ea4010036801ff75c9b2040b20aebfdb992ad0e74f5428e965c429389741711d" gracePeriod=30 Oct 30 01:13:04.895462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:04.895427 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc"] Oct 30 01:13:04.895659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:04.895637 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" containerID="cri-o://7c665c9a6c9aa43356537bb3fa26981c7b4f8a40e54a88b62ba91ed65345a934" gracePeriod=30 Oct 30 01:13:05.391070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.391031 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496"] Oct 30 01:13:05.393883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.393855 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0996ac-76a8-4320-ad1d-7cad3c396659" containerName="istio-proxy" Oct 30 01:13:05.400464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.400443 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:05.413230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.413206 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496"] Oct 30 01:13:05.419342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.418864 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-24d5ce\"" Oct 30 01:13:05.550608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.507861 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-24d5ce-dockercfg-ccv92\"" Oct 30 01:13:05.550608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.509401 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:05.550608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.509732 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxlt\" (UniqueName: \"kubernetes.io/projected/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-kube-api-access-5mxlt\") pod \"payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:05.624549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.616047 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxlt\" (UniqueName: \"kubernetes.io/projected/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-kube-api-access-5mxlt\") pod \"payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:05.624549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.616249 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:05.624549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.618286 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:05.765542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:05.735311 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxlt\" (UniqueName: \"kubernetes.io/projected/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-kube-api-access-5mxlt\") pod \"payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:06.044593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:06.043965 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:06.395640 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:13:06.395604 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3399f18d_1f37_4f43_8b91_0bfff4b22e1d.slice/crio-20df2e91908978139844fbc53b800da98f23bc210ead8b1d88a799ad4f0f1194 WatchSource:0}: Error finding container 20df2e91908978139844fbc53b800da98f23bc210ead8b1d88a799ad4f0f1194: Status 404 returned error can't find the container with id 20df2e91908978139844fbc53b800da98f23bc210ead8b1d88a799ad4f0f1194 Oct 30 01:13:06.397652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:06.397627 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496"] Oct 30 01:13:06.934557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:06.933155 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" event={"ID":"3399f18d-1f37-4f43-8b91-0bfff4b22e1d","Type":"ContainerStarted","Data":"6d46f348e1fa0758b37f95b5909feea7a5473105dca23945f26d863bd8b06fd1"} Oct 30 01:13:06.934557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:06.933222 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" event={"ID":"3399f18d-1f37-4f43-8b91-0bfff4b22e1d","Type":"ContainerStarted","Data":"20df2e91908978139844fbc53b800da98f23bc210ead8b1d88a799ad4f0f1194"} Oct 30 01:13:06.934557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:06.934256 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:06.989005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:06.988941 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" podStartSLOduration=1.988920356 podStartE2EDuration="1.988920356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:06.979891175 +0000 UTC m=+2071.923064372" watchObservedRunningTime="2026-10-30 01:13:06.988920356 +0000 UTC m=+2071.932093550" Oct 30 01:13:08.220905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:08.220884 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:13:08.380577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:08.379165 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-kube-api-access-8p2k5\" (UniqueName: \"kubernetes.io/projected/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-kube-api-access-8p2k5\") pod \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " Oct 30 01:13:08.380577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:08.379403 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-plugins-config-volume\") pod \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\" (UID: \"83dd55cc-49d0-4b6b-a896-95a6e3573ef6\") " Oct 30 01:13:08.381842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:08.381812 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-plugins-config-volume" pod "83dd55cc-49d0-4b6b-a896-95a6e3573ef6" (UID: "83dd55cc-49d0-4b6b-a896-95a6e3573ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:08.401578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:08.401499 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-kube-api-access-8p2k5" pod "83dd55cc-49d0-4b6b-a896-95a6e3573ef6" (UID: "83dd55cc-49d0-4b6b-a896-95a6e3573ef6"). InnerVolumeSpecName "kube-api-access-8p2k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:08.482623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:08.482585 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2k5\" (UniqueName: \"kubernetes.io/projected/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-kube-api-access-8p2k5\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:08.482623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:08.482619 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83dd55cc-49d0-4b6b-a896-95a6e3573ef6-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:09.035554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.033768 4905 generic.go:356] "Generic (PLEG): container finished" podID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" containerID="910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7" exitCode=0 Oct 30 01:13:09.035554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.034911 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" Oct 30 01:13:09.052876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.052769 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" event={"ID":"83dd55cc-49d0-4b6b-a896-95a6e3573ef6","Type":"ContainerDied","Data":"910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7"} Oct 30 01:13:09.052876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.052877 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t" event={"ID":"83dd55cc-49d0-4b6b-a896-95a6e3573ef6","Type":"ContainerDied","Data":"21f12d1053243ff2c92b447527d09b752326272eb1deb315a5a6b4e1920d39b6"} Oct 30 01:13:09.053120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.052911 4905 scope.go:122] "RemoveContainer" containerID="910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7" Oct 30 01:13:09.071367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.070229 4905 scope.go:122] "RemoveContainer" containerID="910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7" Oct 30 01:13:09.071367 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:09.070597 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7\": container with ID starting with 910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7 not found: ID does not exist" containerID="910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7" Oct 30 01:13:09.071367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.070635 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7"} err="failed to get container status \"910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7\": rpc error: code = NotFound desc = could not find container \"910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7\": container with ID starting with 910dc5084d407aea278ecff71ad8cd71fb0c93dce3733a309ebe9f7bc9af8ea7 not found: ID does not exist" Oct 30 01:13:09.111701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.111679 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t"] Oct 30 01:13:09.116906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.116884 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2bf8a-764cfc5d8d-7gk7t"] Oct 30 01:13:09.192080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:09.192050 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" path="/var/lib/kubelet/pods/83dd55cc-49d0-4b6b-a896-95a6e3573ef6/volumes" Oct 30 01:13:10.050401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.050358 4905 generic.go:356] "Generic (PLEG): container finished" podID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerID="ea4010036801ff75c9b2040b20aebfdb992ad0e74f5428e965c429389741711d" exitCode=0 Oct 30 01:13:10.050927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.050502 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" event={"ID":"a5bb9abe-dc1e-4805-9547-63c4606f6570","Type":"ContainerDied","Data":"ea4010036801ff75c9b2040b20aebfdb992ad0e74f5428e965c429389741711d"} Oct 30 01:13:10.053332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.053202 4905 generic.go:356] "Generic (PLEG): container finished" podID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerID="7c665c9a6c9aa43356537bb3fa26981c7b4f8a40e54a88b62ba91ed65345a934" exitCode=0 Oct 30 01:13:10.053332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.053244 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" event={"ID":"dd5362aa-cf21-44ea-8e6d-cd3e540831e9","Type":"ContainerDied","Data":"7c665c9a6c9aa43356537bb3fa26981c7b4f8a40e54a88b62ba91ed65345a934"} Oct 30 01:13:10.257014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.256735 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:13:10.285538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.285485 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:13:10.306811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.306779 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm"] Oct 30 01:13:10.307062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.307016 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" podUID="73d96231-b70d-4e2e-8cd3-5390aee5ab62" containerName="maas-api" containerID="cri-o://89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168" gracePeriod=30 Oct 30 01:13:10.312248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312217 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-token\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-token\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.312379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312299 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-podinfo\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.312379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312333 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-envoy\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.312379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312374 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-kube-api-access-lr629\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-kube-api-access-lr629\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.312620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312408 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istiod-ca-cert\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.312620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312458 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-podinfo\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.312620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312571 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-socket\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.312620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312608 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-token\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-token\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.312835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312639 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-socket\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.312835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312688 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-certs\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.312835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.312781 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-credential-socket\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.314662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.313420 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-socket" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.314662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.313464 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istiod-ca-cert" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:10.314662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.313697 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-certs" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.314662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.313959 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-credential-socket" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.318994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.318301 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-credential-socket\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.318994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.318361 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-data\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.318994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.318532 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-certs\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.318994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.318580 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bb9abe-dc1e-4805-9547-63c4606f6570-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb9abe-dc1e-4805-9547-63c4606f6570-istiod-ca-cert\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.321079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.319751 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-credential-socket" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.321079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.319913 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-data" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.321079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.319966 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-certs" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.321079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.320290 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bb9abe-dc1e-4805-9547-63c4606f6570-istiod-ca-cert" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:10.321773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.321750 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-envoy\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.321915 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.321830 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-kube-api-access-xjtkv\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-kube-api-access-xjtkv\") pod \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\" (UID: \"dd5362aa-cf21-44ea-8e6d-cd3e540831e9\") " Oct 30 01:13:10.322013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.321953 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-data\") pod \"a5bb9abe-dc1e-4805-9547-63c4606f6570\" (UID: \"a5bb9abe-dc1e-4805-9547-63c4606f6570\") " Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322838 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322867 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb9abe-dc1e-4805-9547-63c4606f6570-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322881 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322896 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322909 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322924 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322937 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.322968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.322950 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.324427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.324400 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-socket" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.340656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.339763 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-data" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.364050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.363895 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-podinfo" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:10.364050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.364024 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-podinfo" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:10.366409 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.366368 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-token" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:10.367117 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.367082 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-kube-api-access-xjtkv" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "kube-api-access-xjtkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:10.382828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.382681 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-envoy" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.382828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.382790 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-token" pod "dd5362aa-cf21-44ea-8e6d-cd3e540831e9" (UID: "dd5362aa-cf21-44ea-8e6d-cd3e540831e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:10.391512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.390643 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-kube-api-access-lr629" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "kube-api-access-lr629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:10.391512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.391476 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-envoy" pod "a5bb9abe-dc1e-4805-9547-63c4606f6570" (UID: "a5bb9abe-dc1e-4805-9547-63c4606f6570"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:10.425543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.425459 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.425543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.425498 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.425543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.425507 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.425964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.425936 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr629\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-kube-api-access-lr629\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.425964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.425966 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.426154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.425981 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.426154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.426012 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.426154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.426030 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.426154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.426043 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjtkv\" (UniqueName: \"kubernetes.io/projected/dd5362aa-cf21-44ea-8e6d-cd3e540831e9-kube-api-access-xjtkv\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.426154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.426056 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bb9abe-dc1e-4805-9547-63c4606f6570-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:10.786768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.786724 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:13:10.987727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.983361 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d96231-b70d-4e2e-8cd3-5390aee5ab62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d96231-b70d-4e2e-8cd3-5390aee5ab62-openshift-service-ca\") pod \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " Oct 30 01:13:10.987727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.983511 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-maas-api-tls\") pod \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " Oct 30 01:13:10.987727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.983593 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-metrics-certs\") pod \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " Oct 30 01:13:10.987727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.983632 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d96231-b70d-4e2e-8cd3-5390aee5ab62-kube-api-access-2rjs7\" (UniqueName: \"kubernetes.io/projected/73d96231-b70d-4e2e-8cd3-5390aee5ab62-kube-api-access-2rjs7\") pod \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\" (UID: \"73d96231-b70d-4e2e-8cd3-5390aee5ab62\") " Oct 30 01:13:10.991399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.990849 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d96231-b70d-4e2e-8cd3-5390aee5ab62-openshift-service-ca" pod "73d96231-b70d-4e2e-8cd3-5390aee5ab62" (UID: "73d96231-b70d-4e2e-8cd3-5390aee5ab62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:10.992068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:10.991843 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" podUID="9bc3397b-4c60-40f4-b0d0-950829147631" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:13:11.099367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.099228 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d96231-b70d-4e2e-8cd3-5390aee5ab62-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:11.106255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.106202 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-maas-api-tls" pod "73d96231-b70d-4e2e-8cd3-5390aee5ab62" (UID: "73d96231-b70d-4e2e-8cd3-5390aee5ab62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:11.113886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.106619 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d96231-b70d-4e2e-8cd3-5390aee5ab62-kube-api-access-2rjs7" pod "73d96231-b70d-4e2e-8cd3-5390aee5ab62" (UID: "73d96231-b70d-4e2e-8cd3-5390aee5ab62"). InnerVolumeSpecName "kube-api-access-2rjs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:11.115420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.115392 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-metrics-certs" pod "73d96231-b70d-4e2e-8cd3-5390aee5ab62" (UID: "73d96231-b70d-4e2e-8cd3-5390aee5ab62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:11.124700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.124673 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" Oct 30 01:13:11.125289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.124672 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7" event={"ID":"a5bb9abe-dc1e-4805-9547-63c4606f6570","Type":"ContainerDied","Data":"f4c36b398d0b0a44a5db45e00c191646f966267905ff71f329c6e652f8b533d2"} Oct 30 01:13:11.125398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.125335 4905 scope.go:122] "RemoveContainer" containerID="ea4010036801ff75c9b2040b20aebfdb992ad0e74f5428e965c429389741711d" Oct 30 01:13:11.130104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.130083 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" event={"ID":"dd5362aa-cf21-44ea-8e6d-cd3e540831e9","Type":"ContainerDied","Data":"4a1c6e71c4cb07d903513ddd45dbc3c69bc0d95d0157715c3b8878fb287f2587"} Oct 30 01:13:11.130190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.130170 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc" Oct 30 01:13:11.137171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.136660 4905 generic.go:356] "Generic (PLEG): container finished" podID="73d96231-b70d-4e2e-8cd3-5390aee5ab62" containerID="89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168" exitCode=0 Oct 30 01:13:11.137171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.136702 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" event={"ID":"73d96231-b70d-4e2e-8cd3-5390aee5ab62","Type":"ContainerDied","Data":"89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168"} Oct 30 01:13:11.137171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.136738 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" event={"ID":"73d96231-b70d-4e2e-8cd3-5390aee5ab62","Type":"ContainerDied","Data":"c1badd1a2241bb78a517a685cdb50253d5884dc79d2fa04e9cbd358cef1d95d0"} Oct 30 01:13:11.137171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.136746 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm" Oct 30 01:13:11.166153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.166115 4905 scope.go:122] "RemoveContainer" containerID="7c665c9a6c9aa43356537bb3fa26981c7b4f8a40e54a88b62ba91ed65345a934" Oct 30 01:13:11.191546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.190937 4905 scope.go:122] "RemoveContainer" containerID="89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168" Oct 30 01:13:11.204126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.204091 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:11.204126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.204126 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d96231-b70d-4e2e-8cd3-5390aee5ab62-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:11.204364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.204141 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rjs7\" (UniqueName: \"kubernetes.io/projected/73d96231-b70d-4e2e-8cd3-5390aee5ab62-kube-api-access-2rjs7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:11.221958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.221926 4905 scope.go:122] "RemoveContainer" containerID="89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168" Oct 30 01:13:11.222308 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:11.222286 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168\": container with ID starting with 89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168 not found: ID does not exist" containerID="89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168" Oct 30 01:13:11.222424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.222315 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168"} err="failed to get container status \"89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168\": rpc error: code = NotFound desc = could not find container \"89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168\": container with ID starting with 89ed25ddbb94ea7ef864da9d837f651e78aa898f11c261746b27cbcf49c9e168 not found: ID does not exist" Oct 30 01:13:11.230777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.230748 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7"] Oct 30 01:13:11.234827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.234797 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-96bv7"] Oct 30 01:13:11.270396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.270371 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc"] Oct 30 01:13:11.274595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.274567 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zz5pc"] Oct 30 01:13:11.308877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.308814 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm"] Oct 30 01:13:11.312104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:11.312086 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0680fa-84bf4cff86-vnwmm"] Oct 30 01:13:12.418493 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:12.418437 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:13:13.195650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:13.195614 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d96231-b70d-4e2e-8cd3-5390aee5ab62" path="/var/lib/kubelet/pods/73d96231-b70d-4e2e-8cd3-5390aee5ab62/volumes" Oct 30 01:13:13.196180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:13.196165 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" path="/var/lib/kubelet/pods/a5bb9abe-dc1e-4805-9547-63c4606f6570/volumes" Oct 30 01:13:13.196626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:13.196610 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" path="/var/lib/kubelet/pods/dd5362aa-cf21-44ea-8e6d-cd3e540831e9/volumes" Oct 30 01:13:14.227846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:14.227813 4905 generic.go:356] "Generic (PLEG): container finished" podID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerID="6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4" exitCode=0 Oct 30 01:13:14.228407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:14.227887 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" event={"ID":"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485","Type":"ContainerDied","Data":"6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4"} Oct 30 01:13:14.228407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:14.228316 4905 scope.go:122] "RemoveContainer" containerID="6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4" Oct 30 01:13:14.230079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:14.230051 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:13:14.230161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:14.230124 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:13:14.230161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:14.230157 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:13:15.266270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:15.266229 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" event={"ID":"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485","Type":"ContainerStarted","Data":"75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5"} Oct 30 01:13:15.776764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:15.776688 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:13:15.777063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:15.777042 4905 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 01:13:15.777216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:15.777194 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 01:13:15.878215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:15.878187 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:13:16.059769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.058864 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc3397b-4c60-40f4-b0d0-950829147631-kube-api-access-54r4x\" (UniqueName: \"kubernetes.io/projected/9bc3397b-4c60-40f4-b0d0-950829147631-kube-api-access-54r4x\") pod \"9bc3397b-4c60-40f4-b0d0-950829147631\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " Oct 30 01:13:16.060775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.060753 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc3397b-4c60-40f4-b0d0-950829147631-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc3397b-4c60-40f4-b0d0-950829147631-plugins-config-volume\") pod \"9bc3397b-4c60-40f4-b0d0-950829147631\" (UID: \"9bc3397b-4c60-40f4-b0d0-950829147631\") " Oct 30 01:13:16.061767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.061738 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc3397b-4c60-40f4-b0d0-950829147631-kube-api-access-54r4x" pod "9bc3397b-4c60-40f4-b0d0-950829147631" (UID: "9bc3397b-4c60-40f4-b0d0-950829147631"). InnerVolumeSpecName "kube-api-access-54r4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:16.061943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.061926 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc3397b-4c60-40f4-b0d0-950829147631-plugins-config-volume" pod "9bc3397b-4c60-40f4-b0d0-950829147631" (UID: "9bc3397b-4c60-40f4-b0d0-950829147631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:16.162654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.162607 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54r4x\" (UniqueName: \"kubernetes.io/projected/9bc3397b-4c60-40f4-b0d0-950829147631-kube-api-access-54r4x\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:16.162654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.162632 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc3397b-4c60-40f4-b0d0-950829147631-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:16.275754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.275722 4905 generic.go:356] "Generic (PLEG): container finished" podID="9bc3397b-4c60-40f4-b0d0-950829147631" containerID="876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b" exitCode=0 Oct 30 01:13:16.276211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.275768 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" Oct 30 01:13:16.276211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.275801 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" event={"ID":"9bc3397b-4c60-40f4-b0d0-950829147631","Type":"ContainerDied","Data":"876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b"} Oct 30 01:13:16.276211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.275839 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp" event={"ID":"9bc3397b-4c60-40f4-b0d0-950829147631","Type":"ContainerDied","Data":"00ec752574df1bc122647a8cc46da6aa5115c906bc31e5da8cffeb72b1d65b31"} Oct 30 01:13:16.276211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.275860 4905 scope.go:122] "RemoveContainer" containerID="876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b" Oct 30 01:13:16.287191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.287173 4905 scope.go:122] "RemoveContainer" containerID="876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b" Oct 30 01:13:16.287461 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:16.287445 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b\": container with ID starting with 876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b not found: ID does not exist" containerID="876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b" Oct 30 01:13:16.287509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.287469 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b"} err="failed to get container status \"876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b\": rpc error: code = NotFound desc = could not find container \"876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b\": container with ID starting with 876923b02760cd63907d1d1cc44ca68bfe67bf9b5f397c7423887838e471c89b not found: ID does not exist" Oct 30 01:13:16.354675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.351630 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp"] Oct 30 01:13:16.395563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.394200 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0680fa-6cc457fd95-pjcwp"] Oct 30 01:13:16.756541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.755826 4905 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 01:13:16.756541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:16.755898 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 01:13:17.191913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:17.191839 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc3397b-4c60-40f4-b0d0-950829147631" path="/var/lib/kubelet/pods/9bc3397b-4c60-40f4-b0d0-950829147631/volumes" Oct 30 01:13:17.754543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:17.753820 4905 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 01:13:17.754543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:17.753909 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 01:13:18.172623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:18.171268 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s"] Oct 30 01:13:18.172623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:18.171571 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" containerID="cri-o://11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564" gracePeriod=30 Oct 30 01:13:18.768437 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:18.767225 4905 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 01:13:18.768437 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:18.767294 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 01:13:18.879541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:18.875686 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:13:18.900191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:18.900148 4905 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:13:18.900354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:18.900218 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:13:19.045093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:19.045000 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n"] Oct 30 01:13:19.045912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:19.045879 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" containerID="cri-o://343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80" gracePeriod=30 Oct 30 01:13:19.072686 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:19.072662 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:13:19.754696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:19.754601 4905 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 01:13:19.754696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:19.754662 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 01:13:20.753431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:20.753395 4905 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 01:13:20.754021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:20.753466 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 01:13:21.758379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:21.758348 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:13:21.759465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:21.759446 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:13:23.469566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.469538 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:13:23.519221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.519194 4905 generic.go:356] "Generic (PLEG): container finished" podID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerID="11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564" exitCode=0 Oct 30 01:13:23.519376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.519249 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" Oct 30 01:13:23.519376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.519283 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" event={"ID":"74f3e0d6-a86c-4527-ab11-e6732c123c73","Type":"ContainerDied","Data":"11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564"} Oct 30 01:13:23.519376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.519339 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s" event={"ID":"74f3e0d6-a86c-4527-ab11-e6732c123c73","Type":"ContainerDied","Data":"9156e44bdaa06bac462741da9d8c84f82f2917bf466c40b7c7717588a103c5fd"} Oct 30 01:13:23.519376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.519367 4905 scope.go:122] "RemoveContainer" containerID="11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564" Oct 30 01:13:23.526478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.526457 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-podinfo\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.526604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.526489 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-data\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.526604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.526563 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-kube-api-access-mk85h\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-kube-api-access-mk85h\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.526738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.526723 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-credential-socket\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.526801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.526783 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-certs\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.526993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.526884 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-data" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:23.527094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.527003 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-certs" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:23.527094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.527016 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-credential-socket" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:23.527242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.526980 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-socket\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.527367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.527258 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-socket" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:23.527367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.527290 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-envoy\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.527607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.527589 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f3e0d6-a86c-4527-ab11-e6732c123c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f3e0d6-a86c-4527-ab11-e6732c123c73-istiod-ca-cert\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.527804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.527777 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-token\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-token\") pod \"74f3e0d6-a86c-4527-ab11-e6732c123c73\" (UID: \"74f3e0d6-a86c-4527-ab11-e6732c123c73\") " Oct 30 01:13:23.528115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.528091 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f3e0d6-a86c-4527-ab11-e6732c123c73-istiod-ca-cert" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:23.528596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.528572 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.528700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.528599 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.528700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.528615 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.528700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.528629 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f3e0d6-a86c-4527-ab11-e6732c123c73-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.528700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.528642 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.529497 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.529476 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-podinfo" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:23.530206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.530177 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-kube-api-access-mk85h" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "kube-api-access-mk85h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:23.531004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.530979 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-token" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:23.538351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.538328 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-envoy" pod "74f3e0d6-a86c-4527-ab11-e6732c123c73" (UID: "74f3e0d6-a86c-4527-ab11-e6732c123c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:23.543304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.543280 4905 scope.go:122] "RemoveContainer" containerID="9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8" Oct 30 01:13:23.573810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.573784 4905 scope.go:122] "RemoveContainer" containerID="11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564" Oct 30 01:13:23.574154 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:23.574134 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564\": container with ID starting with 11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564 not found: ID does not exist" containerID="11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564" Oct 30 01:13:23.574251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.574169 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564"} err="failed to get container status \"11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564\": rpc error: code = NotFound desc = could not find container \"11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564\": container with ID starting with 11ca3386e91a0e7c30f12f7ba82baf2143736acf3306d681e06a36c1b81aa564 not found: ID does not exist" Oct 30 01:13:23.574251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.574188 4905 scope.go:122] "RemoveContainer" containerID="9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8" Oct 30 01:13:23.574672 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:23.574652 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8\": container with ID starting with 9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8 not found: ID does not exist" containerID="9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8" Oct 30 01:13:23.574780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.574677 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8"} err="failed to get container status \"9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8\": rpc error: code = NotFound desc = could not find container \"9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8\": container with ID starting with 9e88890b00dff6392d1c98d74d0503b4f79f442e34fdb9e94a7b6d8e5af96bc8 not found: ID does not exist" Oct 30 01:13:23.630593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.630566 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.630688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.630596 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.630688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.630611 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f3e0d6-a86c-4527-ab11-e6732c123c73-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.630688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.630624 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk85h\" (UniqueName: \"kubernetes.io/projected/74f3e0d6-a86c-4527-ab11-e6732c123c73-kube-api-access-mk85h\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:23.870751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.870718 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s"] Oct 30 01:13:23.886680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:23.886652 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-gml4s"] Oct 30 01:13:24.399564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.399285 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:13:24.441492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.441456 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-kube-api-access-q7f97\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-kube-api-access-q7f97\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.441684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.441510 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-socket\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.441684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.441605 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-token\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-token\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.441811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.441752 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-data\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.441811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.441778 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-socket" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:24.441927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.441855 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-credential-socket\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.441983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.441947 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-podinfo\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.442054 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.442030 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-data" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:24.442472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.442448 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-certs\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.442865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.442839 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-certs" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:24.443011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.442929 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-envoy\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.443107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.443033 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istiod-ca-cert\") pod \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\" (UID: \"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7\") " Oct 30 01:13:24.446303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.446264 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-credential-socket" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:24.446468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.446324 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-token" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:24.446770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.446735 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istiod-ca-cert" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:24.447244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.447206 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-kube-api-access-q7f97" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "kube-api-access-q7f97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:24.447594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.447574 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-podinfo" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:24.449906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.449879 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7f97\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-kube-api-access-q7f97\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.449906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.449911 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.450087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.449926 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.450087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.449938 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.451777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.449952 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.451884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.451808 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.451884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.451827 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.451884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.451875 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.451884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.451056 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-envoy" pod "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" (UID: "ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:24.528311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.528274 4905 generic.go:356] "Generic (PLEG): container finished" podID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerID="343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80" exitCode=0 Oct 30 01:13:24.528793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.528321 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" Oct 30 01:13:24.528793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.528366 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" event={"ID":"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7","Type":"ContainerDied","Data":"343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80"} Oct 30 01:13:24.528793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.528422 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n" event={"ID":"ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7","Type":"ContainerDied","Data":"63c8a766e040ee0eb22ae8fda52bf72bb6ecb0dcccbfaf4425188d5daeaa59d6"} Oct 30 01:13:24.528793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.528454 4905 scope.go:122] "RemoveContainer" containerID="343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80" Oct 30 01:13:24.540450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.540431 4905 scope.go:122] "RemoveContainer" containerID="343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80" Oct 30 01:13:24.540752 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:24.540731 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80\": container with ID starting with 343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80 not found: ID does not exist" containerID="343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80" Oct 30 01:13:24.540846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.540757 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80"} err="failed to get container status \"343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80\": rpc error: code = NotFound desc = could not find container \"343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80\": container with ID starting with 343bd52d2e0837a7b529b4a5939c9ec4fea3985adcfe5f334f2738f768483f80 not found: ID does not exist" Oct 30 01:13:24.553078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.553050 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:24.613753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.613661 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n"] Oct 30 01:13:24.617765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:24.617737 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-dsl4n"] Oct 30 01:13:25.194594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:25.194560 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" path="/var/lib/kubelet/pods/74f3e0d6-a86c-4527-ab11-e6732c123c73/volumes" Oct 30 01:13:25.195286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:25.195259 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" path="/var/lib/kubelet/pods/ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7/volumes" Oct 30 01:13:29.523411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:29.523379 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb"] Oct 30 01:13:32.667954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.667905 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74"] Oct 30 01:13:32.669918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.669884 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" Oct 30 01:13:32.669918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.669908 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.669951 4905 cpu_manager.go:414] "containerMap: removing container" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.669959 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.669983 4905 cpu_manager.go:414] "containerMap: removing container" podUID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" containerName="payload-processing" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.669991 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" containerName="payload-processing" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670042 4905 cpu_manager.go:414] "containerMap: removing container" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670050 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670058 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9bc3397b-4c60-40f4-b0d0-950829147631" containerName="payload-pre-processing" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670065 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc3397b-4c60-40f4-b0d0-950829147631" containerName="payload-pre-processing" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670081 4905 cpu_manager.go:414] "containerMap: removing container" podUID="73d96231-b70d-4e2e-8cd3-5390aee5ab62" containerName="maas-api" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670088 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d96231-b70d-4e2e-8cd3-5390aee5ab62" containerName="maas-api" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670099 4905 cpu_manager.go:414] "containerMap: removing container" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670106 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" Oct 30 01:13:32.670135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670141 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670148 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670324 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5362aa-cf21-44ea-8e6d-cd3e540831e9" containerName="istio-proxy" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670342 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="83dd55cc-49d0-4b6b-a896-95a6e3573ef6" containerName="payload-processing" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670357 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670368 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc3397b-4c60-40f4-b0d0-950829147631" containerName="payload-pre-processing" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670380 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d96231-b70d-4e2e-8cd3-5390aee5ab62" containerName="maas-api" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670396 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f3e0d6-a86c-4527-ab11-e6732c123c73" containerName="istio-proxy" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670409 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8bf617-86cd-4e4f-ac1b-a67508dcd1c7" containerName="istio-proxy" Oct 30 01:13:32.670862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.670420 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bb9abe-dc1e-4805-9547-63c4606f6570" containerName="istio-proxy" Oct 30 01:13:32.674460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.674439 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.694296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.694266 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74"] Oct 30 01:13:32.707579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.703215 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca324d\"/\"default-dockercfg-hwwxz\"" Oct 30 01:13:32.707579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.703277 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca324d\"/\"sub-test-model-ca324d-kserve-self-signed-certs\"" Oct 30 01:13:32.707579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.705130 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca324d\"/\"openshift-service-ca.crt\"" Oct 30 01:13:32.707579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.705219 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca324d\"/\"kube-root-ca.crt\"" Oct 30 01:13:32.817922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.817877 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj9jb\" (UniqueName: \"kubernetes.io/projected/8144991b-f4ea-4314-962e-376ff644a7d3-kube-api-access-mj9jb\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.818100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.817984 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8144991b-f4ea-4314-962e-376ff644a7d3-tls-certs\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.818100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.818084 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-home\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.818190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.818147 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-dshm\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.818296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.818281 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-tmp-dir\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.818339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.818317 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-model-cache\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.922802 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-tmp-dir\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.922871 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-model-cache\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.922927 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj9jb\" (UniqueName: \"kubernetes.io/projected/8144991b-f4ea-4314-962e-376ff644a7d3-kube-api-access-mj9jb\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.922994 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8144991b-f4ea-4314-962e-376ff644a7d3-tls-certs\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.923100 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-home\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.923172 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-dshm\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.924487 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-tmp-dir\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.925626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.924784 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-model-cache\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.934564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.933735 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-home\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.946543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.941864 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-dshm\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.954540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.949192 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj9jb\" (UniqueName: \"kubernetes.io/projected/8144991b-f4ea-4314-962e-376ff644a7d3-kube-api-access-mj9jb\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:32.976543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:32.965556 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8144991b-f4ea-4314-962e-376ff644a7d3-tls-certs\") pod \"sub-test-model-ca324d-kserve-75fdf44b7-ssq74\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:33.012786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:33.011834 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:33.331087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:33.331058 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74"] Oct 30 01:13:33.331220 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:13:33.331189 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8144991b_f4ea_4314_962e_376ff644a7d3.slice/crio-40aca10e05dec995e1bffad15cdcc491248ed6029aa02a8eb26bb6fe50eaf7df WatchSource:0}: Error finding container 40aca10e05dec995e1bffad15cdcc491248ed6029aa02a8eb26bb6fe50eaf7df: Status 404 returned error can't find the container with id 40aca10e05dec995e1bffad15cdcc491248ed6029aa02a8eb26bb6fe50eaf7df Oct 30 01:13:33.610241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:33.610146 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" event={"ID":"8144991b-f4ea-4314-962e-376ff644a7d3","Type":"ContainerStarted","Data":"1f15df15dcdde245046b19abceac281e0df119d150902ed05f00f31cfbb4139e"} Oct 30 01:13:33.610241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:33.610189 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" event={"ID":"8144991b-f4ea-4314-962e-376ff644a7d3","Type":"ContainerStarted","Data":"40aca10e05dec995e1bffad15cdcc491248ed6029aa02a8eb26bb6fe50eaf7df"} Oct 30 01:13:33.655232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:33.655170 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" podStartSLOduration=1.655152194 podStartE2EDuration="1.655152194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:33.651607983 +0000 UTC m=+2098.594781179" watchObservedRunningTime="2026-10-30 01:13:33.655152194 +0000 UTC m=+2098.598325388" Oct 30 01:13:34.013584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:34.013556 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:34.524149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:34.524110 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" podUID="21efb889-95cc-411b-9da7-071f0203eb8a" containerName="payload-processing" containerID="cri-o://bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e" gracePeriod=25 Oct 30 01:13:35.801970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:35.801937 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:13:35.803631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:35.803609 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:13:40.095874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.095835 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224"] Oct 30 01:13:40.101718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.101693 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.123169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.123141 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224"] Oct 30 01:13:40.131008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.130983 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-24d5ce-serving-cert\"" Oct 30 01:13:40.149075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.149040 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46decaca-579e-43a0-a628-4d41627365ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.149297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.149277 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.149495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.149472 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.149693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.149562 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grp26\" (UniqueName: \"kubernetes.io/projected/46decaca-579e-43a0-a628-4d41627365ed-kube-api-access-grp26\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.250888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.250854 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.251089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.251030 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.251089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.251080 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grp26\" (UniqueName: \"kubernetes.io/projected/46decaca-579e-43a0-a628-4d41627365ed-kube-api-access-grp26\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.251234 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:40.251109 4905 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-serving-cert: secret "maas-api-e2e-shared-a-w2-24d5ce-serving-cert" not found Oct 30 01:13:40.251234 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:40.251187 4905 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls podName:46decaca-579e-43a0-a628-4d41627365ed nodeName:}" failed. No retries permitted until 2026-10-30 01:13:40.751166987 +0000 UTC m=+2105.694340173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls") pod "maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" (UID: "46decaca-579e-43a0-a628-4d41627365ed") : secret "maas-api-e2e-shared-a-w2-24d5ce-serving-cert" not found Oct 30 01:13:40.251361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.251339 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46decaca-579e-43a0-a628-4d41627365ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.251926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.251901 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46decaca-579e-43a0-a628-4d41627365ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.254285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.254255 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.284500 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.283493 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grp26\" (UniqueName: \"kubernetes.io/projected/46decaca-579e-43a0-a628-4d41627365ed-kube-api-access-grp26\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.759607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.759575 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:40.762193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:40.762162 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:41.022165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:41.022071 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:41.379202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:41.370285 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224"] Oct 30 01:13:41.685911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:41.685837 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" event={"ID":"46decaca-579e-43a0-a628-4d41627365ed","Type":"ContainerStarted","Data":"83b1ffd6c970c6671b043c8147cc1165ad5762b73510513bc42b3f1ab2b2de67"} Oct 30 01:13:42.416610 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:42.416567 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:13:42.695661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:42.695574 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" event={"ID":"46decaca-579e-43a0-a628-4d41627365ed","Type":"ContainerStarted","Data":"087bd7d21efbf1cc474a8bcb129e6963d5b7de8d200b8e92522364c5b1f61d63"} Oct 30 01:13:42.695828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:42.695670 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:42.741175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:42.741121 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" podStartSLOduration=2.44390878 podStartE2EDuration="2.741106364s" totalImagesPullingTime="297.197584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:42.739604225 +0000 UTC m=+2107.682777417" watchObservedRunningTime="2026-10-30 01:13:42.741106364 +0000 UTC m=+2107.684279588" Oct 30 01:13:43.013665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:43.013636 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:43.028472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:43.028444 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:43.723568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:43.723508 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:13:44.409705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:44.409663 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" podUID="21efb889-95cc-411b-9da7-071f0203eb8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:13:44.533713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:44.513908 4905 scope.go:122] "RemoveContainer" containerID="082f7eda2fe1948bded6d130ac1da1569927b677648d5af28418cfaac57abe8c" Oct 30 01:13:48.712829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:48.712748 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:13:52.084146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:52.084112 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc"] Oct 30 01:13:54.406846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.406812 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" podUID="21efb889-95cc-411b-9da7-071f0203eb8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:13:54.840036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.840007 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:13:54.922759 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.922716 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21efb889-95cc-411b-9da7-071f0203eb8a-kube-api-access-sbqzb\" (UniqueName: \"kubernetes.io/projected/21efb889-95cc-411b-9da7-071f0203eb8a-kube-api-access-sbqzb\") pod \"21efb889-95cc-411b-9da7-071f0203eb8a\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " Oct 30 01:13:54.923036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.923015 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21efb889-95cc-411b-9da7-071f0203eb8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21efb889-95cc-411b-9da7-071f0203eb8a-plugins-config-volume\") pod \"21efb889-95cc-411b-9da7-071f0203eb8a\" (UID: \"21efb889-95cc-411b-9da7-071f0203eb8a\") " Oct 30 01:13:54.924418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.924388 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21efb889-95cc-411b-9da7-071f0203eb8a-plugins-config-volume" pod "21efb889-95cc-411b-9da7-071f0203eb8a" (UID: "21efb889-95cc-411b-9da7-071f0203eb8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:54.926715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.926693 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21efb889-95cc-411b-9da7-071f0203eb8a-kube-api-access-sbqzb" pod "21efb889-95cc-411b-9da7-071f0203eb8a" (UID: "21efb889-95cc-411b-9da7-071f0203eb8a"). InnerVolumeSpecName "kube-api-access-sbqzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:54.997936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.997902 4905 generic.go:356] "Generic (PLEG): container finished" podID="21efb889-95cc-411b-9da7-071f0203eb8a" containerID="bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e" exitCode=0 Oct 30 01:13:54.998154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.997945 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" Oct 30 01:13:54.998154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.997990 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" event={"ID":"21efb889-95cc-411b-9da7-071f0203eb8a","Type":"ContainerDied","Data":"bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e"} Oct 30 01:13:54.998154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.998048 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb" event={"ID":"21efb889-95cc-411b-9da7-071f0203eb8a","Type":"ContainerDied","Data":"3657af1f5d4be7675daf363ee577c0962f7791490232d0604f6cc1a649a31bec"} Oct 30 01:13:54.998154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:54.998069 4905 scope.go:122] "RemoveContainer" containerID="bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e" Oct 30 01:13:55.011181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:55.011152 4905 scope.go:122] "RemoveContainer" containerID="bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e" Oct 30 01:13:55.011596 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:13:55.011573 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e\": container with ID starting with bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e not found: ID does not exist" containerID="bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e" Oct 30 01:13:55.011676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:55.011602 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e"} err="failed to get container status \"bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e\": rpc error: code = NotFound desc = could not find container \"bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e\": container with ID starting with bd88655b37253a2b101e3a6e6fbe0cb39a60f7ecfa9ad18f53a2815018761e7e not found: ID does not exist" Oct 30 01:13:55.053110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:55.053059 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21efb889-95cc-411b-9da7-071f0203eb8a-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:55.053383 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:55.053363 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqzb\" (UniqueName: \"kubernetes.io/projected/21efb889-95cc-411b-9da7-071f0203eb8a-kube-api-access-sbqzb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:13:55.077129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:55.077086 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb"] Oct 30 01:13:55.085330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:55.085288 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d54608-7b4dc74ccb-7fpcb"] Oct 30 01:13:55.192536 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:55.192486 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21efb889-95cc-411b-9da7-071f0203eb8a" path="/var/lib/kubelet/pods/21efb889-95cc-411b-9da7-071f0203eb8a/volumes" Oct 30 01:13:57.084602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:57.084564 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" podUID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" containerName="payload-processing" containerID="cri-o://6517d993a1009a225d0aa953463683ed47846d8197756ae0f0be73233dfba809" gracePeriod=25 Oct 30 01:13:59.160021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:13:59.159977 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" podUID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:14:00.665435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:00.665377 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s"] Oct 30 01:14:00.666367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:00.666334 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" containerID="cri-o://1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879" gracePeriod=30 Oct 30 01:14:04.595883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:04.595846 4905 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:14:04.596292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:04.595914 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:14:05.998785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:05.998756 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:14:06.054089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054051 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-podinfo\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054116 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-kube-api-access-qr8kb\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-kube-api-access-qr8kb\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054190 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-credential-socket\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054298 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-token\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-token\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054337 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-socket\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054368 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef45997-65ed-4532-8fb2-ce882d7ec527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef45997-65ed-4532-8fb2-ce882d7ec527-istiod-ca-cert\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054436 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-envoy\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054546 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-credential-socket" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:06.054704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054601 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-data\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.054704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054619 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-socket" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:06.054704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054653 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-certs\") pod \"7ef45997-65ed-4532-8fb2-ce882d7ec527\" (UID: \"7ef45997-65ed-4532-8fb2-ce882d7ec527\") " Oct 30 01:14:06.055212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.054892 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-data" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:06.055351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.055283 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef45997-65ed-4532-8fb2-ce882d7ec527-istiod-ca-cert" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:06.055432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.055356 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-certs" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:06.056404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.056307 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.056404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.056335 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.056404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.056354 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.056404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.056368 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef45997-65ed-4532-8fb2-ce882d7ec527-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.056404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.056382 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.057098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.057074 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-podinfo" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:14:06.057251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.057235 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-kube-api-access-qr8kb" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "kube-api-access-qr8kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:06.057357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.057342 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-envoy" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:06.059289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.059263 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-token" pod "7ef45997-65ed-4532-8fb2-ce882d7ec527" (UID: "7ef45997-65ed-4532-8fb2-ce882d7ec527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:06.157766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.157719 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.157766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.157754 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.157766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.157769 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8kb\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-kube-api-access-qr8kb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.158237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.157784 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef45997-65ed-4532-8fb2-ce882d7ec527-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:06.158237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.158095 4905 generic.go:356] "Generic (PLEG): container finished" podID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerID="1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879" exitCode=0 Oct 30 01:14:06.158237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.158193 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" event={"ID":"7ef45997-65ed-4532-8fb2-ce882d7ec527","Type":"ContainerDied","Data":"1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879"} Oct 30 01:14:06.158416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.158258 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" event={"ID":"7ef45997-65ed-4532-8fb2-ce882d7ec527","Type":"ContainerDied","Data":"7a9e13b0593774960008202a6cffa4e701df463c90cc38c19e4de7abfd97ce69"} Oct 30 01:14:06.158416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.158294 4905 scope.go:122] "RemoveContainer" containerID="1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879" Oct 30 01:14:06.158571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.158500 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s" Oct 30 01:14:06.174401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.174372 4905 scope.go:122] "RemoveContainer" containerID="3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068" Oct 30 01:14:06.185689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.185667 4905 scope.go:122] "RemoveContainer" containerID="1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879" Oct 30 01:14:06.186011 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:14:06.185984 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879\": container with ID starting with 1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879 not found: ID does not exist" containerID="1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879" Oct 30 01:14:06.186142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.186024 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879"} err="failed to get container status \"1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879\": rpc error: code = NotFound desc = could not find container \"1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879\": container with ID starting with 1c8f73ee18469b2cf84a9e2a89085b0e799e140990628704d0f237b41beb1879 not found: ID does not exist" Oct 30 01:14:06.186142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.186044 4905 scope.go:122] "RemoveContainer" containerID="3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068" Oct 30 01:14:06.186371 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:14:06.186352 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068\": container with ID starting with 3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068 not found: ID does not exist" containerID="3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068" Oct 30 01:14:06.186438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.186386 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068"} err="failed to get container status \"3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068\": rpc error: code = NotFound desc = could not find container \"3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068\": container with ID starting with 3c2f429353302c4c6b41f8080e44473455d212a8ba108bbd2d0ef4eabf516068 not found: ID does not exist" Oct 30 01:14:06.219324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.219233 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s"] Oct 30 01:14:06.225025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:06.224993 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-d9p8s"] Oct 30 01:14:07.192037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:07.192004 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" path="/var/lib/kubelet/pods/7ef45997-65ed-4532-8fb2-ce882d7ec527/volumes" Oct 30 01:14:09.160078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:09.160033 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" podUID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:14:12.453863 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:14:12.453824 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:14:17.259121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.259087 4905 generic.go:356] "Generic (PLEG): container finished" podID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" containerID="6517d993a1009a225d0aa953463683ed47846d8197756ae0f0be73233dfba809" exitCode=0 Oct 30 01:14:17.259548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.259166 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" event={"ID":"275b80bd-da5b-40b2-8f10-3fddaf2aebab","Type":"ContainerDied","Data":"6517d993a1009a225d0aa953463683ed47846d8197756ae0f0be73233dfba809"} Oct 30 01:14:17.395373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.395348 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:14:17.562168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.562130 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275b80bd-da5b-40b2-8f10-3fddaf2aebab-kube-api-access-zk6qd\" (UniqueName: \"kubernetes.io/projected/275b80bd-da5b-40b2-8f10-3fddaf2aebab-kube-api-access-zk6qd\") pod \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " Oct 30 01:14:17.562358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.562344 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275b80bd-da5b-40b2-8f10-3fddaf2aebab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275b80bd-da5b-40b2-8f10-3fddaf2aebab-plugins-config-volume\") pod \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\" (UID: \"275b80bd-da5b-40b2-8f10-3fddaf2aebab\") " Oct 30 01:14:17.562752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.562727 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275b80bd-da5b-40b2-8f10-3fddaf2aebab-plugins-config-volume" pod "275b80bd-da5b-40b2-8f10-3fddaf2aebab" (UID: "275b80bd-da5b-40b2-8f10-3fddaf2aebab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:17.563014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.562987 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275b80bd-da5b-40b2-8f10-3fddaf2aebab-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:17.564489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.564464 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275b80bd-da5b-40b2-8f10-3fddaf2aebab-kube-api-access-zk6qd" pod "275b80bd-da5b-40b2-8f10-3fddaf2aebab" (UID: "275b80bd-da5b-40b2-8f10-3fddaf2aebab"). InnerVolumeSpecName "kube-api-access-zk6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:17.664417 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:17.664323 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk6qd\" (UniqueName: \"kubernetes.io/projected/275b80bd-da5b-40b2-8f10-3fddaf2aebab-kube-api-access-zk6qd\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:18.271165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:18.271137 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" Oct 30 01:14:18.271680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:18.271171 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc" event={"ID":"275b80bd-da5b-40b2-8f10-3fddaf2aebab","Type":"ContainerDied","Data":"4b840348007774137afccc69d84a9e9ccb28a94fd099e7936821b0a8a603ae5b"} Oct 30 01:14:18.271680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:18.271233 4905 scope.go:122] "RemoveContainer" containerID="6517d993a1009a225d0aa953463683ed47846d8197756ae0f0be73233dfba809" Oct 30 01:14:18.332576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:18.332541 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc"] Oct 30 01:14:18.344226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:18.344187 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-286d36-54c688684b-z64mc"] Oct 30 01:14:19.193402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:19.193370 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" path="/var/lib/kubelet/pods/275b80bd-da5b-40b2-8f10-3fddaf2aebab/volumes" Oct 30 01:14:20.747187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.747155 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g"] Oct 30 01:14:20.747950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.747927 4905 cpu_manager.go:414] "containerMap: removing container" podUID="21efb889-95cc-411b-9da7-071f0203eb8a" containerName="payload-processing" Oct 30 01:14:20.748161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748135 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21efb889-95cc-411b-9da7-071f0203eb8a" containerName="payload-processing" Oct 30 01:14:20.748283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748176 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" Oct 30 01:14:20.748283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748185 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" Oct 30 01:14:20.748283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748207 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" Oct 30 01:14:20.748283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748214 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" Oct 30 01:14:20.748283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748263 4905 cpu_manager.go:414] "containerMap: removing container" podUID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" containerName="payload-processing" Oct 30 01:14:20.748283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748270 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" containerName="payload-processing" Oct 30 01:14:20.748663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748530 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="21efb889-95cc-411b-9da7-071f0203eb8a" containerName="payload-processing" Oct 30 01:14:20.748663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748545 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" Oct 30 01:14:20.748663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748555 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef45997-65ed-4532-8fb2-ce882d7ec527" containerName="istio-proxy" Oct 30 01:14:20.748663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.748572 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="275b80bd-da5b-40b2-8f10-3fddaf2aebab" containerName="payload-processing" Oct 30 01:14:20.753949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.753928 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.776849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.776747 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-cfe8f0-openshift-default-dockercfg-4g4ql\"" Oct 30 01:14:20.796384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.793382 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g"] Oct 30 01:14:20.920044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920014 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-certs\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920098 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-envoy\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920221 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17922593-835e-4bd3-9e76-7510d501815f-istiod-ca-cert\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920311 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17922593-835e-4bd3-9e76-7510d501815f-istio-podinfo\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920346 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920379 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klc6n\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-kube-api-access-klc6n\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920466 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-data\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920537 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-istio-token\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:20.920909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:20.920596 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-credential-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.036544 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-certs\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.036696 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-envoy\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.036774 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17922593-835e-4bd3-9e76-7510d501815f-istiod-ca-cert\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.036877 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17922593-835e-4bd3-9e76-7510d501815f-istio-podinfo\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.036916 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.036954 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klc6n\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-kube-api-access-klc6n\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.037050 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-data\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.037087 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-istio-token\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.037144 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-credential-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.037685 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-credential-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.043601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.037960 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-certs\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.052674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.051811 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.052674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.051874 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-data\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.052674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.052512 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17922593-835e-4bd3-9e76-7510d501815f-istiod-ca-cert\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.081875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.079292 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-envoy\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.081875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.079866 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17922593-835e-4bd3-9e76-7510d501815f-istio-podinfo\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.100541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.091957 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klc6n\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-kube-api-access-klc6n\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.120658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.116047 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-istio-token\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.400412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.399809 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:21.805808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.804236 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:21.805808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.804327 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:21.805808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.804372 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:21.816095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:21.816056 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g"] Oct 30 01:14:22.478994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:22.476475 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" event={"ID":"17922593-835e-4bd3-9e76-7510d501815f","Type":"ContainerStarted","Data":"d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c"} Oct 30 01:14:22.478994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:22.476578 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" event={"ID":"17922593-835e-4bd3-9e76-7510d501815f","Type":"ContainerStarted","Data":"a8f7657138d549ab042de8e4dea3005376538df3259a1b46fdb430c23c3388e3"} Oct 30 01:14:22.561477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:22.561407 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" podStartSLOduration=2.561387921 podStartE2EDuration="2.561387921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:22.541802474 +0000 UTC m=+2147.484975668" watchObservedRunningTime="2026-10-30 01:14:22.561387921 +0000 UTC m=+2147.504561120" Oct 30 01:14:23.411722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:23.405312 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:23.426551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:23.423788 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:23.501545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:23.498131 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:23.501545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:23.500582 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:14:27.175766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.175724 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs"] Oct 30 01:14:27.186571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.183206 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs"] Oct 30 01:14:27.186571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.183492 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:27.201558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.201506 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-cfe8f0\"" Oct 30 01:14:27.213981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.213958 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-cfe8f0-dockercfg-d52rl\"" Oct 30 01:14:27.223681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.223655 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv"] Oct 30 01:14:27.287547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.277704 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv"] Oct 30 01:14:27.287547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.277819 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.362993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.362950 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:27.366733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.366697 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4fhq\" (UniqueName: \"kubernetes.io/projected/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-kube-api-access-d4fhq\") pod \"payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:27.468601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.468424 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4w7\" (UniqueName: \"kubernetes.io/projected/b4110fc9-997e-46ed-9968-9b39240e7afb-kube-api-access-wj4w7\") pod \"payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.472150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.468799 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4110fc9-997e-46ed-9968-9b39240e7afb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.472150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.468898 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:27.472150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.469012 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4fhq\" (UniqueName: \"kubernetes.io/projected/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-kube-api-access-d4fhq\") pod \"payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:27.483548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.474349 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:27.607502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.607464 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4fhq\" (UniqueName: \"kubernetes.io/projected/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-kube-api-access-d4fhq\") pod \"payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:27.617578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.615169 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4w7\" (UniqueName: \"kubernetes.io/projected/b4110fc9-997e-46ed-9968-9b39240e7afb-kube-api-access-wj4w7\") pod \"payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.617578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.615323 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4110fc9-997e-46ed-9968-9b39240e7afb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.617578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.616490 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4110fc9-997e-46ed-9968-9b39240e7afb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.679407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.678951 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4w7\" (UniqueName: \"kubernetes.io/projected/b4110fc9-997e-46ed-9968-9b39240e7afb-kube-api-access-wj4w7\") pod \"payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.761945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.761909 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:27.803989 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:27.803961 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:28.107015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.106981 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv"] Oct 30 01:14:28.113712 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:14:28.113684 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4110fc9_997e_46ed_9968_9b39240e7afb.slice/crio-ddd32e6fbab808377cf87450009b617e20b7b1a6c3c5173391899ffdb980fdab WatchSource:0}: Error finding container ddd32e6fbab808377cf87450009b617e20b7b1a6c3c5173391899ffdb980fdab: Status 404 returned error can't find the container with id ddd32e6fbab808377cf87450009b617e20b7b1a6c3c5173391899ffdb980fdab Oct 30 01:14:28.169000 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:14:28.168967 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593b152c_46a1_4f22_b9b0_bc47a7ad2aa0.slice/crio-2804bcca2548f63b341ea143047263003e7a4a8ba6b789905378ca2da334bcae WatchSource:0}: Error finding container 2804bcca2548f63b341ea143047263003e7a4a8ba6b789905378ca2da334bcae: Status 404 returned error can't find the container with id 2804bcca2548f63b341ea143047263003e7a4a8ba6b789905378ca2da334bcae Oct 30 01:14:28.170730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.170704 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs"] Oct 30 01:14:28.621924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.621890 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" event={"ID":"b4110fc9-997e-46ed-9968-9b39240e7afb","Type":"ContainerStarted","Data":"a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4"} Oct 30 01:14:28.622407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.621949 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" event={"ID":"b4110fc9-997e-46ed-9968-9b39240e7afb","Type":"ContainerStarted","Data":"ddd32e6fbab808377cf87450009b617e20b7b1a6c3c5173391899ffdb980fdab"} Oct 30 01:14:28.622407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.622083 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:28.623897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.623868 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" event={"ID":"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0","Type":"ContainerStarted","Data":"779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a"} Oct 30 01:14:28.624048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.623917 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" event={"ID":"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0","Type":"ContainerStarted","Data":"2804bcca2548f63b341ea143047263003e7a4a8ba6b789905378ca2da334bcae"} Oct 30 01:14:28.624048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.623948 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:28.662664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.662602 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" podStartSLOduration=1.6625578060000001 podStartE2EDuration="1.662557806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:28.662108035 +0000 UTC m=+2153.605281234" watchObservedRunningTime="2026-10-30 01:14:28.662557806 +0000 UTC m=+2153.605730993" Oct 30 01:14:28.710712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:28.710584 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" podStartSLOduration=1.710565898 podStartE2EDuration="1.710565898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:28.70794978 +0000 UTC m=+2153.651122977" watchObservedRunningTime="2026-10-30 01:14:28.710565898 +0000 UTC m=+2153.653739092" Oct 30 01:14:32.353876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:32.353836 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt"] Oct 30 01:14:32.354430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:32.354139 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" podUID="9250ee6e-2572-4c59-a49e-24af53cd813a" containerName="maas-api" containerID="cri-o://ea2be39dcb73132a09717370c3421eb5abf3afec17c9ba5ed8552ca80112b0e0" gracePeriod=30 Oct 30 01:14:32.786604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:32.782753 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:14:32.786604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:32.782826 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:32.928121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:32.928086 4905 generic.go:356] "Generic (PLEG): container finished" podID="9250ee6e-2572-4c59-a49e-24af53cd813a" containerID="ea2be39dcb73132a09717370c3421eb5abf3afec17c9ba5ed8552ca80112b0e0" exitCode=0 Oct 30 01:14:32.928604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:32.928573 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" event={"ID":"9250ee6e-2572-4c59-a49e-24af53cd813a","Type":"ContainerDied","Data":"ea2be39dcb73132a09717370c3421eb5abf3afec17c9ba5ed8552ca80112b0e0"} Oct 30 01:14:32.998824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:32.998785 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:14:33.063061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.062966 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-maas-api-tls\") pod \"9250ee6e-2572-4c59-a49e-24af53cd813a\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " Oct 30 01:14:33.063384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.063359 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-metrics-certs\") pod \"9250ee6e-2572-4c59-a49e-24af53cd813a\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " Oct 30 01:14:33.064597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.064561 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9250ee6e-2572-4c59-a49e-24af53cd813a-openshift-service-ca" pod "9250ee6e-2572-4c59-a49e-24af53cd813a" (UID: "9250ee6e-2572-4c59-a49e-24af53cd813a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:33.064713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.064665 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9250ee6e-2572-4c59-a49e-24af53cd813a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9250ee6e-2572-4c59-a49e-24af53cd813a-openshift-service-ca\") pod \"9250ee6e-2572-4c59-a49e-24af53cd813a\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " Oct 30 01:14:33.066179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.065124 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9250ee6e-2572-4c59-a49e-24af53cd813a-kube-api-access-mgf7w\" (UniqueName: \"kubernetes.io/projected/9250ee6e-2572-4c59-a49e-24af53cd813a-kube-api-access-mgf7w\") pod \"9250ee6e-2572-4c59-a49e-24af53cd813a\" (UID: \"9250ee6e-2572-4c59-a49e-24af53cd813a\") " Oct 30 01:14:33.066831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.066798 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-metrics-certs" pod "9250ee6e-2572-4c59-a49e-24af53cd813a" (UID: "9250ee6e-2572-4c59-a49e-24af53cd813a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:14:33.067792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.067753 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:33.071750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.067810 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9250ee6e-2572-4c59-a49e-24af53cd813a-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:33.078051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.078023 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-maas-api-tls" pod "9250ee6e-2572-4c59-a49e-24af53cd813a" (UID: "9250ee6e-2572-4c59-a49e-24af53cd813a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:14:33.080382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.080356 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9250ee6e-2572-4c59-a49e-24af53cd813a-kube-api-access-mgf7w" pod "9250ee6e-2572-4c59-a49e-24af53cd813a" (UID: "9250ee6e-2572-4c59-a49e-24af53cd813a"). InnerVolumeSpecName "kube-api-access-mgf7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:33.173329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.173298 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgf7w\" (UniqueName: \"kubernetes.io/projected/9250ee6e-2572-4c59-a49e-24af53cd813a-kube-api-access-mgf7w\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:33.173329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.173327 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9250ee6e-2572-4c59-a49e-24af53cd813a-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:33.939149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.939118 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" Oct 30 01:14:33.939754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.939150 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt" event={"ID":"9250ee6e-2572-4c59-a49e-24af53cd813a","Type":"ContainerDied","Data":"27530e7969350483c0ca2806a13c54f96f867d4fdea1b6c88464bac36d5cb16e"} Oct 30 01:14:33.939754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.939227 4905 scope.go:122] "RemoveContainer" containerID="ea2be39dcb73132a09717370c3421eb5abf3afec17c9ba5ed8552ca80112b0e0" Oct 30 01:14:33.942016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.941992 4905 generic.go:356] "Generic (PLEG): container finished" podID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerID="04486989f9ddf11a9951f190939bd89dc151ad9433ccecfeb3f313941f4191ef" exitCode=0 Oct 30 01:14:33.942158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.942051 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" event={"ID":"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a","Type":"ContainerDied","Data":"04486989f9ddf11a9951f190939bd89dc151ad9433ccecfeb3f313941f4191ef"} Oct 30 01:14:33.942492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.942475 4905 scope.go:122] "RemoveContainer" containerID="04486989f9ddf11a9951f190939bd89dc151ad9433ccecfeb3f313941f4191ef" Oct 30 01:14:33.945040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.945002 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:33.945139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.945112 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:33.945191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:33.945156 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:34.015666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:34.015635 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt"] Oct 30 01:14:34.021732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:34.021704 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bc6092-6b85f94b44-sc2lt"] Oct 30 01:14:34.967936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:34.967665 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" event={"ID":"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a","Type":"ContainerStarted","Data":"67f9a9da8e619e589393cba62b0798fc0869de6d27450f275117d8c169d13068"} Oct 30 01:14:35.100695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:35.100659 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:14:35.100872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:35.100807 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 01:14:35.100872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:35.100857 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 01:14:35.194080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:35.194042 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9250ee6e-2572-4c59-a49e-24af53cd813a" path="/var/lib/kubelet/pods/9250ee6e-2572-4c59-a49e-24af53cd813a/volumes" Oct 30 01:14:36.100909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:36.100863 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 01:14:36.101321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:36.100929 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 01:14:37.101766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:37.100676 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 01:14:37.101766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:37.100727 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 01:14:38.100573 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:38.100539 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 01:14:38.100783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:38.100594 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 01:14:39.100972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:39.100938 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 01:14:39.101396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:39.101006 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 01:14:39.659442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:39.659405 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:14:39.659965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:39.659943 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:14:40.086542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.086322 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww"] Oct 30 01:14:40.093551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.091632 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" containerID="cri-o://148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1" gracePeriod=30 Oct 30 01:14:40.106347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.106318 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 01:14:40.106883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.106859 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 01:14:40.633595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.633491 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr"] Oct 30 01:14:40.634723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.634697 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9250ee6e-2572-4c59-a49e-24af53cd813a" containerName="maas-api" Oct 30 01:14:40.634723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.634723 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9250ee6e-2572-4c59-a49e-24af53cd813a" containerName="maas-api" Oct 30 01:14:40.634930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.634923 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9250ee6e-2572-4c59-a49e-24af53cd813a" containerName="maas-api" Oct 30 01:14:40.639418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.639395 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.661372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.661349 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b312f7-openshift-default-dockercfg-t8sp7\"" Oct 30 01:14:40.672579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.667626 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr"] Oct 30 01:14:40.832973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.832939 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-token\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.832995 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-data\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.833059 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-envoy\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.833179 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istiod-ca-cert\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.833229 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-credential-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.833343 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-podinfo\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.833396 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc98r\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-kube-api-access-sc98r\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.833440 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.833668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.833648 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-certs\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935039 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-certs\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935202 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-token\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935243 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-data\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935271 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-envoy\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935486 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935461 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-certs\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935513 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istiod-ca-cert\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935580 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-credential-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935628 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-podinfo\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935638 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-data\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935679 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc98r\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-kube-api-access-sc98r\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935873 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935717 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.935937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.935892 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-credential-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.936341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.936309 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istiod-ca-cert\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.936453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.936441 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.938179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.938150 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-envoy\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.938666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.938645 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-podinfo\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.958867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.958840 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc98r\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-kube-api-access-sc98r\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:40.959012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:40.958989 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-token\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:41.104773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.104738 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:14:41.105131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.105107 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:14:41.105893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.105875 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:14:41.264153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.262836 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:41.820050 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:14:41.820023 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff5b4b8_e670_4715_b021_f2334afdbe4f.slice/crio-e64168473e46c1ed13c48002b337cf17f74f002214b5889cd5c5bd82ee728e3b WatchSource:0}: Error finding container e64168473e46c1ed13c48002b337cf17f74f002214b5889cd5c5bd82ee728e3b: Status 404 returned error can't find the container with id e64168473e46c1ed13c48002b337cf17f74f002214b5889cd5c5bd82ee728e3b Oct 30 01:14:41.822428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.822404 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr"] Oct 30 01:14:41.823420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.823388 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:41.823509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.823453 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:41.823509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:41.823485 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:14:42.053541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:42.053189 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" event={"ID":"2ff5b4b8-e670-4715-b021-f2334afdbe4f","Type":"ContainerStarted","Data":"58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09"} Oct 30 01:14:42.053541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:42.053255 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" event={"ID":"2ff5b4b8-e670-4715-b021-f2334afdbe4f","Type":"ContainerStarted","Data":"e64168473e46c1ed13c48002b337cf17f74f002214b5889cd5c5bd82ee728e3b"} Oct 30 01:14:42.103913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:42.103808 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podStartSLOduration=2.103791136 podStartE2EDuration="2.103791136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:42.100193684 +0000 UTC m=+2167.043366879" watchObservedRunningTime="2026-10-30 01:14:42.103791136 +0000 UTC m=+2167.046964338" Oct 30 01:14:42.264378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:42.264337 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:42.265997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:42.265968 4905 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 01:14:42.266129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:42.266024 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 01:14:42.417107 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:14:42.417016 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:14:43.264362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:43.264324 4905 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 01:14:43.264611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:43.264396 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 01:14:44.265363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:44.265322 4905 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 01:14:44.266021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:44.265985 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 01:14:44.656400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:44.655819 4905 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:14:44.656400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:44.655882 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:14:45.266541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.265956 4905 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 01:14:45.266541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.266023 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 01:14:45.522778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.522751 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656020 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f280e47e-36f4-4451-a458-311662858ecd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f280e47e-36f4-4451-a458-311662858ecd-istiod-ca-cert\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656109 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-istio-token\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-istio-token\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656173 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-certs\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656220 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-kube-api-access-smxjn\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-kube-api-access-smxjn\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656310 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-data\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656484 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-credential-socket\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656579 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f280e47e-36f4-4451-a458-311662858ecd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f280e47e-36f4-4451-a458-311662858ecd-istio-podinfo\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656637 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-envoy\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.656677 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-socket\") pod \"f280e47e-36f4-4451-a458-311662858ecd\" (UID: \"f280e47e-36f4-4451-a458-311662858ecd\") " Oct 30 01:14:45.663555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.662634 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f280e47e-36f4-4451-a458-311662858ecd-istiod-ca-cert" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:45.664231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.663619 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-socket" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:45.664231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.663743 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-certs" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:45.664231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.663985 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-credential-socket" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:45.664684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.664660 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-data" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:45.666486 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.666419 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-istio-token" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:45.667118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.666882 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-kube-api-access-smxjn" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "kube-api-access-smxjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:45.667333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.667300 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f280e47e-36f4-4451-a458-311662858ecd-istio-podinfo" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:14:45.667494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.667467 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-envoy" pod "f280e47e-36f4-4451-a458-311662858ecd" (UID: "f280e47e-36f4-4451-a458-311662858ecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758809 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f280e47e-36f4-4451-a458-311662858ecd-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758861 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758877 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758890 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f280e47e-36f4-4451-a458-311662858ecd-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758903 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758916 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758929 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxjn\" (UniqueName: \"kubernetes.io/projected/f280e47e-36f4-4451-a458-311662858ecd-kube-api-access-smxjn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758942 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.759073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:45.758955 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f280e47e-36f4-4451-a458-311662858ecd-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:14:46.101172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.101075 4905 generic.go:356] "Generic (PLEG): container finished" podID="f280e47e-36f4-4451-a458-311662858ecd" containerID="148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1" exitCode=0 Oct 30 01:14:46.101172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.101141 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" Oct 30 01:14:46.101172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.101145 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" event={"ID":"f280e47e-36f4-4451-a458-311662858ecd","Type":"ContainerDied","Data":"148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1"} Oct 30 01:14:46.101456 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.101216 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww" event={"ID":"f280e47e-36f4-4451-a458-311662858ecd","Type":"ContainerDied","Data":"a4ce67d265a55ceb2be84fe9b76f83ad842c2ccd4247567051b3a548f21c3f77"} Oct 30 01:14:46.101456 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.101245 4905 scope.go:122] "RemoveContainer" containerID="148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1" Oct 30 01:14:46.119230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.119201 4905 scope.go:122] "RemoveContainer" containerID="13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd" Oct 30 01:14:46.164604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.157738 4905 scope.go:122] "RemoveContainer" containerID="148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1" Oct 30 01:14:46.168583 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:14:46.168536 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1\": container with ID starting with 148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1 not found: ID does not exist" containerID="148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1" Oct 30 01:14:46.168770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.168589 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1"} err="failed to get container status \"148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1\": rpc error: code = NotFound desc = could not find container \"148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1\": container with ID starting with 148ca7262a4c3a1728604ff000be1554c7364126b79d3aafce955ba6203dace1 not found: ID does not exist" Oct 30 01:14:46.168770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.168616 4905 scope.go:122] "RemoveContainer" containerID="13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd" Oct 30 01:14:46.172576 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:14:46.169845 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd\": container with ID starting with 13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd not found: ID does not exist" containerID="13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd" Oct 30 01:14:46.172576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.169884 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd"} err="failed to get container status \"13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd\": rpc error: code = NotFound desc = could not find container \"13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd\": container with ID starting with 13578c1893f930ae7b107bfe663482c25bc995a5464074786cde181180194bcd not found: ID does not exist" Oct 30 01:14:46.183391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.183361 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww"] Oct 30 01:14:46.187172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.187146 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-j2vww"] Oct 30 01:14:46.264244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.264194 4905 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 01:14:46.264459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:46.264254 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 01:14:47.193374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.193344 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f280e47e-36f4-4451-a458-311662858ecd" path="/var/lib/kubelet/pods/f280e47e-36f4-4451-a458-311662858ecd/volumes" Oct 30 01:14:47.193790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.193740 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7"] Oct 30 01:14:47.194416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.194399 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" Oct 30 01:14:47.194490 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.194416 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" Oct 30 01:14:47.194602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.194590 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" Oct 30 01:14:47.198188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.198172 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.208318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.208297 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7"] Oct 30 01:14:47.217494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.215897 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-cb31e3-dockercfg-b2xdr\"" Oct 30 01:14:47.217494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.216642 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-cb31e3\"" Oct 30 01:14:47.479416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.413640 4905 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 01:14:47.479416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.413701 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 01:14:47.479416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.455385 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdcb\" (UniqueName: \"kubernetes.io/projected/1df68de3-3599-4f4c-9e25-7e85f942467d-kube-api-access-8xdcb\") pod \"payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.479416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.455619 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df68de3-3599-4f4c-9e25-7e85f942467d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.519714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.517823 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw"] Oct 30 01:14:47.519714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.519083 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" Oct 30 01:14:47.519714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.519105 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" Oct 30 01:14:47.519714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.519287 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f280e47e-36f4-4451-a458-311662858ecd" containerName="istio-proxy" Oct 30 01:14:47.525428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.525406 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:47.550473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.550442 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw"] Oct 30 01:14:47.575047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.575009 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7hc\" (UniqueName: \"kubernetes.io/projected/1e90d226-6961-4cd5-8132-77c006205f67-kube-api-access-gq7hc\") pod \"payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:47.575207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.575098 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdcb\" (UniqueName: \"kubernetes.io/projected/1df68de3-3599-4f4c-9e25-7e85f942467d-kube-api-access-8xdcb\") pod \"payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.575207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.575168 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df68de3-3599-4f4c-9e25-7e85f942467d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.575335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.575276 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e90d226-6961-4cd5-8132-77c006205f67-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:47.576466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.576433 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df68de3-3599-4f4c-9e25-7e85f942467d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.596269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.596240 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdcb\" (UniqueName: \"kubernetes.io/projected/1df68de3-3599-4f4c-9e25-7e85f942467d-kube-api-access-8xdcb\") pod \"payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.676378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.676339 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e90d226-6961-4cd5-8132-77c006205f67-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:47.676583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.676498 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7hc\" (UniqueName: \"kubernetes.io/projected/1e90d226-6961-4cd5-8132-77c006205f67-kube-api-access-gq7hc\") pod \"payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:47.677099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.677079 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e90d226-6961-4cd5-8132-77c006205f67-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:47.697283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.697262 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7hc\" (UniqueName: \"kubernetes.io/projected/1e90d226-6961-4cd5-8132-77c006205f67-kube-api-access-gq7hc\") pod \"payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:47.819788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.819758 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:47.857592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:47.855820 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:48.196848 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:14:48.196814 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e90d226_6961_4cd5_8132_77c006205f67.slice/crio-a6a0797d17cb9272d518cdebb5db046d6cc03239bf8ab5d7d7148afab1222f3d WatchSource:0}: Error finding container a6a0797d17cb9272d518cdebb5db046d6cc03239bf8ab5d7d7148afab1222f3d: Status 404 returned error can't find the container with id a6a0797d17cb9272d518cdebb5db046d6cc03239bf8ab5d7d7148afab1222f3d Oct 30 01:14:48.197817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:48.197790 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw"] Oct 30 01:14:48.207830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:48.207805 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7"] Oct 30 01:14:48.236677 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:14:48.236605 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df68de3_3599_4f4c_9e25_7e85f942467d.slice/crio-700d5d8fe247ef1c639ad318855ce867d401a9f70176e2d869f56f3d561c8fca WatchSource:0}: Error finding container 700d5d8fe247ef1c639ad318855ce867d401a9f70176e2d869f56f3d561c8fca: Status 404 returned error can't find the container with id 700d5d8fe247ef1c639ad318855ce867d401a9f70176e2d869f56f3d561c8fca Oct 30 01:14:48.280213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:48.279728 4905 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 01:14:48.280213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:48.279784 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 01:14:49.129431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.129390 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" event={"ID":"1df68de3-3599-4f4c-9e25-7e85f942467d","Type":"ContainerStarted","Data":"37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e"} Oct 30 01:14:49.129663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.129539 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" event={"ID":"1df68de3-3599-4f4c-9e25-7e85f942467d","Type":"ContainerStarted","Data":"700d5d8fe247ef1c639ad318855ce867d401a9f70176e2d869f56f3d561c8fca"} Oct 30 01:14:49.129663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.129578 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:14:49.131313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.131284 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" event={"ID":"1e90d226-6961-4cd5-8132-77c006205f67","Type":"ContainerStarted","Data":"cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736"} Oct 30 01:14:49.131460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.131336 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" event={"ID":"1e90d226-6961-4cd5-8132-77c006205f67","Type":"ContainerStarted","Data":"a6a0797d17cb9272d518cdebb5db046d6cc03239bf8ab5d7d7148afab1222f3d"} Oct 30 01:14:49.131460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.131383 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:14:49.174151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.174100 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" podStartSLOduration=2.174081893 podStartE2EDuration="2.174081893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:49.170138217 +0000 UTC m=+2174.113311413" watchObservedRunningTime="2026-10-30 01:14:49.174081893 +0000 UTC m=+2174.117255085" Oct 30 01:14:49.219996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.219948 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" podStartSLOduration=2.219931782 podStartE2EDuration="2.219931782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:49.215378568 +0000 UTC m=+2174.158551762" watchObservedRunningTime="2026-10-30 01:14:49.219931782 +0000 UTC m=+2174.163104978" Oct 30 01:14:49.270030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.270000 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:49.270547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.270506 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:14:49.271434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:14:49.271412 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:15:00.147162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.147123 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl"] Oct 30 01:15:00.174914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.174881 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:15:00.176403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.176380 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.187752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.187726 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:15:00.189459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.189423 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8"] Oct 30 01:15:00.196700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.196677 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.197864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.197846 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf"] Oct 30 01:15:00.215082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.215046 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl"] Oct 30 01:15:00.215223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.215211 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.219495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.219316 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg"] Oct 30 01:15:00.276402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.276374 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:15:00.283088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.283054 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8"] Oct 30 01:15:00.283385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.283370 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf"] Oct 30 01:15:00.283547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.283314 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.289381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.288450 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg"] Oct 30 01:15:00.300964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.300733 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c"] Oct 30 01:15:00.301620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.301240 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.310060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.309583 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg"] Oct 30 01:15:00.310060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.309624 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj"] Oct 30 01:15:00.311168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.311116 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.318611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.318588 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k"] Oct 30 01:15:00.323376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.323352 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.327707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.327685 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl"] Oct 30 01:15:00.331380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.331353 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.343798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.343775 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.343907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.343819 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg"] Oct 30 01:15:00.349028 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.349008 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c"] Oct 30 01:15:00.354057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.354036 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj"] Oct 30 01:15:00.355666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.355640 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.356004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.355972 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b168dc55-dfba-45d0-b281-80390072df92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.356113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.356014 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgr2\" (UniqueName: \"kubernetes.io/projected/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-kube-api-access-2wgr2\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.356113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.356106 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbdq\" (UniqueName: \"kubernetes.io/projected/b168dc55-dfba-45d0-b281-80390072df92-kube-api-access-dmbdq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.356271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.356252 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff5453-b37c-496b-8abc-e95ff86338df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.356338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.356319 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blmn\" (UniqueName: \"kubernetes.io/projected/f1ff5453-b37c-496b-8abc-e95ff86338df-kube-api-access-4blmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.360507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.360485 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k"] Oct 30 01:15:00.365290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.365264 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl"] Oct 30 01:15:00.457498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457405 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a4faf7-b736-44ef-bc82-195af1bf0af8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.457696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457511 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e0c08a-049e-484e-aab6-ddeeed75c2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.457696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457642 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2p6\" (UniqueName: \"kubernetes.io/projected/33374e9d-1513-4645-a044-d979e11ac979-kube-api-access-ct2p6\") pod \"maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.457802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457724 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47grb\" (UniqueName: \"kubernetes.io/projected/c40fd573-8d64-425e-9134-054f7440e187-kube-api-access-47grb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.457802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457781 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b168dc55-dfba-45d0-b281-80390072df92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.457892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457813 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79ws\" (UniqueName: \"kubernetes.io/projected/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-kube-api-access-t79ws\") pod \"maas-api-key-cleanup-29888715-lr27c\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.457892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457841 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgr2\" (UniqueName: \"kubernetes.io/projected/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-kube-api-access-2wgr2\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.457999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457865 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33374e9d-1513-4645-a044-d979e11ac979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.457999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.457975 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbdq\" (UniqueName: \"kubernetes.io/projected/b168dc55-dfba-45d0-b281-80390072df92-kube-api-access-dmbdq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.458101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.458025 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vpx\" (UniqueName: \"kubernetes.io/projected/f379fc6d-07de-491f-8c4b-210a84498a91-kube-api-access-q4vpx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.458101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.458055 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-lr27c\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.458180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.458161 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qff\" (UniqueName: \"kubernetes.io/projected/54a4faf7-b736-44ef-bc82-195af1bf0af8-kube-api-access-g8qff\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.458223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.458194 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40fd573-8d64-425e-9134-054f7440e187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.458278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.458222 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff5453-b37c-496b-8abc-e95ff86338df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.458735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.458697 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blmn\" (UniqueName: \"kubernetes.io/projected/f1ff5453-b37c-496b-8abc-e95ff86338df-kube-api-access-4blmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.459033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.458994 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b168dc55-dfba-45d0-b281-80390072df92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.459163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.459100 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.459482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.459234 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff5453-b37c-496b-8abc-e95ff86338df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.459482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.459234 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzkg\" (UniqueName: \"kubernetes.io/projected/31e0c08a-049e-484e-aab6-ddeeed75c2f1-kube-api-access-cqzkg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.459482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.459283 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f379fc6d-07de-491f-8c4b-210a84498a91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.460003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.459972 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.479060 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.479037 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blmn\" (UniqueName: \"kubernetes.io/projected/f1ff5453-b37c-496b-8abc-e95ff86338df-kube-api-access-4blmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.479423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.479395 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbdq\" (UniqueName: \"kubernetes.io/projected/b168dc55-dfba-45d0-b281-80390072df92-kube-api-access-dmbdq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.480727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.480705 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgr2\" (UniqueName: \"kubernetes.io/projected/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-kube-api-access-2wgr2\") pod \"maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.504898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.504876 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:00.521353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.521329 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569389 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79ws\" (UniqueName: \"kubernetes.io/projected/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-kube-api-access-t79ws\") pod \"maas-api-key-cleanup-29888715-lr27c\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569435 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33374e9d-1513-4645-a044-d979e11ac979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569490 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vpx\" (UniqueName: \"kubernetes.io/projected/f379fc6d-07de-491f-8c4b-210a84498a91-kube-api-access-q4vpx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569510 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-lr27c\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569628 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qff\" (UniqueName: \"kubernetes.io/projected/54a4faf7-b736-44ef-bc82-195af1bf0af8-kube-api-access-g8qff\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569662 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40fd573-8d64-425e-9134-054f7440e187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569865 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzkg\" (UniqueName: \"kubernetes.io/projected/31e0c08a-049e-484e-aab6-ddeeed75c2f1-kube-api-access-cqzkg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569895 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f379fc6d-07de-491f-8c4b-210a84498a91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569951 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a4faf7-b736-44ef-bc82-195af1bf0af8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.569988 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e0c08a-049e-484e-aab6-ddeeed75c2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.570191 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2p6\" (UniqueName: \"kubernetes.io/projected/33374e9d-1513-4645-a044-d979e11ac979-kube-api-access-ct2p6\") pod \"maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.570291 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47grb\" (UniqueName: \"kubernetes.io/projected/c40fd573-8d64-425e-9134-054f7440e187-kube-api-access-47grb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.572060 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e0c08a-049e-484e-aab6-ddeeed75c2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.572086 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40fd573-8d64-425e-9134-054f7440e187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.572508 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33374e9d-1513-4645-a044-d979e11ac979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.572817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.572739 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a4faf7-b736-44ef-bc82-195af1bf0af8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.573936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.573411 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-lr27c\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.573936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.573720 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f379fc6d-07de-491f-8c4b-210a84498a91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.593757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.592654 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzkg\" (UniqueName: \"kubernetes.io/projected/31e0c08a-049e-484e-aab6-ddeeed75c2f1-kube-api-access-cqzkg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.596011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.595110 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79ws\" (UniqueName: \"kubernetes.io/projected/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-kube-api-access-t79ws\") pod \"maas-api-key-cleanup-29888715-lr27c\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.596292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.596270 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vpx\" (UniqueName: \"kubernetes.io/projected/f379fc6d-07de-491f-8c4b-210a84498a91-kube-api-access-q4vpx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.597115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.597076 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47grb\" (UniqueName: \"kubernetes.io/projected/c40fd573-8d64-425e-9134-054f7440e187-kube-api-access-47grb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.598351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.598316 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2p6\" (UniqueName: \"kubernetes.io/projected/33374e9d-1513-4645-a044-d979e11ac979-kube-api-access-ct2p6\") pod \"maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.598961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.598901 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qff\" (UniqueName: \"kubernetes.io/projected/54a4faf7-b736-44ef-bc82-195af1bf0af8-kube-api-access-g8qff\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.602972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.602585 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:00.637548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.637207 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:00.658711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.656663 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:00.665821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.665800 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:00.681786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.681589 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:00.690103 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.689717 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:00.737770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.710248 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:00.870981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:00.870870 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl"] Oct 30 01:15:01.200201 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:15:01.200174 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ff5453_b37c_496b_8abc_e95ff86338df.slice/crio-51aa4e06c3fc903ea6637868034eb219efc487d84fe4fb10aa993e969d4e9db0 WatchSource:0}: Error finding container 51aa4e06c3fc903ea6637868034eb219efc487d84fe4fb10aa993e969d4e9db0: Status 404 returned error can't find the container with id 51aa4e06c3fc903ea6637868034eb219efc487d84fe4fb10aa993e969d4e9db0 Oct 30 01:15:01.264576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.260507 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8"] Oct 30 01:15:01.269134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.269065 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg"] Oct 30 01:15:01.283321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.283297 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf"] Oct 30 01:15:01.293031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.293008 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c"] Oct 30 01:15:01.307805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.307776 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg"] Oct 30 01:15:01.324586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.324552 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerStarted","Data":"51aa4e06c3fc903ea6637868034eb219efc487d84fe4fb10aa993e969d4e9db0"} Oct 30 01:15:01.327155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.327123 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" event={"ID":"b168dc55-dfba-45d0-b281-80390072df92","Type":"ContainerStarted","Data":"98daacdb0af8f81b3641e68b77986fbb16f8cf45a3ce2360e0abb468d628957f"} Oct 30 01:15:01.327924 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:15:01.327896 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40fd573_8d64_425e_9134_054f7440e187.slice/crio-560d266bf2b2a2a70cbcd704c019f8cfaac1a03d34c600d0efe5cfbe9401aafa WatchSource:0}: Error finding container 560d266bf2b2a2a70cbcd704c019f8cfaac1a03d34c600d0efe5cfbe9401aafa: Status 404 returned error can't find the container with id 560d266bf2b2a2a70cbcd704c019f8cfaac1a03d34c600d0efe5cfbe9401aafa Oct 30 01:15:01.339178 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:15:01.339155 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4b96d8_ffc3_4e55_83de_4a3b8d216b56.slice/crio-574a27cedabd82dc14c783ee2ce202b01e7ea5182cd59d10b416d1c1080838f7 WatchSource:0}: Error finding container 574a27cedabd82dc14c783ee2ce202b01e7ea5182cd59d10b416d1c1080838f7: Status 404 returned error can't find the container with id 574a27cedabd82dc14c783ee2ce202b01e7ea5182cd59d10b416d1c1080838f7 Oct 30 01:15:01.414302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.414173 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj"] Oct 30 01:15:01.466158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.466079 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl"] Oct 30 01:15:01.505320 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:15:01.505292 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e0c08a_049e_484e_aab6_ddeeed75c2f1.slice/crio-11552f7aa7337a896c5271e6e93fa248593650d056f873afdbb9fe716dcc365a WatchSource:0}: Error finding container 11552f7aa7337a896c5271e6e93fa248593650d056f873afdbb9fe716dcc365a: Status 404 returned error can't find the container with id 11552f7aa7337a896c5271e6e93fa248593650d056f873afdbb9fe716dcc365a Oct 30 01:15:01.506688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:01.506663 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k"] Oct 30 01:15:02.342788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.342755 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" event={"ID":"33374e9d-1513-4645-a044-d979e11ac979","Type":"ContainerStarted","Data":"cf4ab4f8a7d942c32fcdee5110c9198ebf752809a1d163533e137fa0650b4852"} Oct 30 01:15:02.343487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.343455 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" event={"ID":"33374e9d-1513-4645-a044-d979e11ac979","Type":"ContainerStarted","Data":"05d7747d42b9afad847f149f58b99170f42ecbf4043a6cafc3820e4b15f5043a"} Oct 30 01:15:02.346761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.346730 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" event={"ID":"c40fd573-8d64-425e-9134-054f7440e187","Type":"ContainerStarted","Data":"0123daeeb57c9e2b0a2408909edeb3044e451a23150974313807850442fd7da3"} Oct 30 01:15:02.346889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.346787 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" event={"ID":"c40fd573-8d64-425e-9134-054f7440e187","Type":"ContainerStarted","Data":"560d266bf2b2a2a70cbcd704c019f8cfaac1a03d34c600d0efe5cfbe9401aafa"} Oct 30 01:15:02.348687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.348663 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" event={"ID":"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56","Type":"ContainerStarted","Data":"db6180ca63a7120426804b9d1c5e74a6a928bfb53c02cbbc05b9ef6411e73d20"} Oct 30 01:15:02.348797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.348712 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" event={"ID":"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56","Type":"ContainerStarted","Data":"574a27cedabd82dc14c783ee2ce202b01e7ea5182cd59d10b416d1c1080838f7"} Oct 30 01:15:02.350478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.350455 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerStarted","Data":"c8d3a879223893ce1332fdcff7e2b000ff0fddd6fc69d02a4b258f1102c390a8"} Oct 30 01:15:02.352267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.352244 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" event={"ID":"54a4faf7-b736-44ef-bc82-195af1bf0af8","Type":"ContainerStarted","Data":"85dc04caba2a9b34db4935d5f815a8c687d0e2741123afd1ce7e7bbac9dc96f2"} Oct 30 01:15:02.352374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.352287 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" event={"ID":"54a4faf7-b736-44ef-bc82-195af1bf0af8","Type":"ContainerStarted","Data":"f078323e1924f363f870b7b663651ea9882b9bf9ec192cd9bdc871e9d4a47eaa"} Oct 30 01:15:02.353939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.353912 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" event={"ID":"9a049cef-bb00-46d6-85e9-7fbd280b3d0e","Type":"ContainerStarted","Data":"70344984be68d10efe887a489c63097c2c5d52436aa244b2b7251c0249fdbf75"} Oct 30 01:15:02.355627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.355602 4905 generic.go:356] "Generic (PLEG): container finished" podID="b168dc55-dfba-45d0-b281-80390072df92" containerID="5a5f7b61059ce5cc58d7ee2fdccecda7650d65be07a0bb9eb26630f437799ee6" exitCode=60 Oct 30 01:15:02.355754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.355698 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" event={"ID":"b168dc55-dfba-45d0-b281-80390072df92","Type":"ContainerDied","Data":"5a5f7b61059ce5cc58d7ee2fdccecda7650d65be07a0bb9eb26630f437799ee6"} Oct 30 01:15:02.355906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.355888 4905 scope.go:122] "RemoveContainer" containerID="5a5f7b61059ce5cc58d7ee2fdccecda7650d65be07a0bb9eb26630f437799ee6" Oct 30 01:15:02.357641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.357582 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" event={"ID":"31e0c08a-049e-484e-aab6-ddeeed75c2f1","Type":"ContainerStarted","Data":"f9d8cc66e2e7c664715affbe6b2fa8d6e2d8a9bf61454d736464470ee207900a"} Oct 30 01:15:02.357641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.357623 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" event={"ID":"31e0c08a-049e-484e-aab6-ddeeed75c2f1","Type":"ContainerStarted","Data":"11552f7aa7337a896c5271e6e93fa248593650d056f873afdbb9fe716dcc365a"} Oct 30 01:15:02.366238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.365598 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" event={"ID":"f379fc6d-07de-491f-8c4b-210a84498a91","Type":"ContainerStarted","Data":"cf35f41d3fc2c687ebbc66c7a9f571bc239c880a1c330d4b6e96585c35f447e4"} Oct 30 01:15:02.366238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.365646 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" event={"ID":"f379fc6d-07de-491f-8c4b-210a84498a91","Type":"ContainerStarted","Data":"f59c2dd56e77af3941b696590dd366bc3337c69a5a4acd1d97d332e1fd0df64a"} Oct 30 01:15:02.385328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.385288 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" podStartSLOduration=2.385276935 podStartE2EDuration="2.385276935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.383723305 +0000 UTC m=+2187.326896499" watchObservedRunningTime="2026-10-30 01:15:02.385276935 +0000 UTC m=+2187.328450128" Oct 30 01:15:02.436554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.436482 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" podStartSLOduration=2.436465879 podStartE2EDuration="2.436465879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.423286285 +0000 UTC m=+2187.366459477" watchObservedRunningTime="2026-10-30 01:15:02.436465879 +0000 UTC m=+2187.379639073" Oct 30 01:15:02.464571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.464508 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" podStartSLOduration=2.4644943550000002 podStartE2EDuration="2.464494355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.463189351 +0000 UTC m=+2187.406362548" watchObservedRunningTime="2026-10-30 01:15:02.464494355 +0000 UTC m=+2187.407667550" Oct 30 01:15:02.547225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.547173 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" podStartSLOduration=2.547159302 podStartE2EDuration="2.547159302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.546005785 +0000 UTC m=+2187.489178979" watchObservedRunningTime="2026-10-30 01:15:02.547159302 +0000 UTC m=+2187.490332494" Oct 30 01:15:02.587161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.587117 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" podStartSLOduration=2.587103185 podStartE2EDuration="2.587103185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.586540608 +0000 UTC m=+2187.529713798" watchObservedRunningTime="2026-10-30 01:15:02.587103185 +0000 UTC m=+2187.530276397" Oct 30 01:15:02.630366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.630316 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" podStartSLOduration=2.630298465 podStartE2EDuration="2.630298465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.629337222 +0000 UTC m=+2187.572510416" watchObservedRunningTime="2026-10-30 01:15:02.630298465 +0000 UTC m=+2187.573471660" Oct 30 01:15:02.673912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.673863 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" podStartSLOduration=2.6738500800000002 podStartE2EDuration="2.67385008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.669634915 +0000 UTC m=+2187.612808120" watchObservedRunningTime="2026-10-30 01:15:02.67385008 +0000 UTC m=+2187.617023271" Oct 30 01:15:02.713182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:02.713135 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" podStartSLOduration=2.713120037 podStartE2EDuration="2.713120037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.709021865 +0000 UTC m=+2187.652195068" watchObservedRunningTime="2026-10-30 01:15:02.713120037 +0000 UTC m=+2187.656293255" Oct 30 01:15:03.379021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.378985 4905 generic.go:356] "Generic (PLEG): container finished" podID="33374e9d-1513-4645-a044-d979e11ac979" containerID="cf4ab4f8a7d942c32fcdee5110c9198ebf752809a1d163533e137fa0650b4852" exitCode=60 Oct 30 01:15:03.379431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.379068 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" event={"ID":"33374e9d-1513-4645-a044-d979e11ac979","Type":"ContainerDied","Data":"cf4ab4f8a7d942c32fcdee5110c9198ebf752809a1d163533e137fa0650b4852"} Oct 30 01:15:03.379431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.379389 4905 scope.go:122] "RemoveContainer" containerID="cf4ab4f8a7d942c32fcdee5110c9198ebf752809a1d163533e137fa0650b4852" Oct 30 01:15:03.381254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.381202 4905 generic.go:356] "Generic (PLEG): container finished" podID="c40fd573-8d64-425e-9134-054f7440e187" containerID="0123daeeb57c9e2b0a2408909edeb3044e451a23150974313807850442fd7da3" exitCode=60 Oct 30 01:15:03.381254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.381240 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" event={"ID":"c40fd573-8d64-425e-9134-054f7440e187","Type":"ContainerDied","Data":"0123daeeb57c9e2b0a2408909edeb3044e451a23150974313807850442fd7da3"} Oct 30 01:15:03.381583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.381561 4905 scope.go:122] "RemoveContainer" containerID="0123daeeb57c9e2b0a2408909edeb3044e451a23150974313807850442fd7da3" Oct 30 01:15:03.384421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.384403 4905 generic.go:356] "Generic (PLEG): container finished" podID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerID="db6180ca63a7120426804b9d1c5e74a6a928bfb53c02cbbc05b9ef6411e73d20" exitCode=60 Oct 30 01:15:03.384583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.384566 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" event={"ID":"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56","Type":"ContainerDied","Data":"db6180ca63a7120426804b9d1c5e74a6a928bfb53c02cbbc05b9ef6411e73d20"} Oct 30 01:15:03.384832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.384811 4905 scope.go:122] "RemoveContainer" containerID="db6180ca63a7120426804b9d1c5e74a6a928bfb53c02cbbc05b9ef6411e73d20" Oct 30 01:15:03.386926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.386904 4905 generic.go:356] "Generic (PLEG): container finished" podID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerID="c8d3a879223893ce1332fdcff7e2b000ff0fddd6fc69d02a4b258f1102c390a8" exitCode=60 Oct 30 01:15:03.387014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.386984 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerDied","Data":"c8d3a879223893ce1332fdcff7e2b000ff0fddd6fc69d02a4b258f1102c390a8"} Oct 30 01:15:03.387226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.387206 4905 scope.go:122] "RemoveContainer" containerID="c8d3a879223893ce1332fdcff7e2b000ff0fddd6fc69d02a4b258f1102c390a8" Oct 30 01:15:03.389338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.389278 4905 generic.go:356] "Generic (PLEG): container finished" podID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerID="85dc04caba2a9b34db4935d5f815a8c687d0e2741123afd1ce7e7bbac9dc96f2" exitCode=60 Oct 30 01:15:03.389338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.389355 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" event={"ID":"54a4faf7-b736-44ef-bc82-195af1bf0af8","Type":"ContainerDied","Data":"85dc04caba2a9b34db4935d5f815a8c687d0e2741123afd1ce7e7bbac9dc96f2"} Oct 30 01:15:03.389708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.389694 4905 scope.go:122] "RemoveContainer" containerID="85dc04caba2a9b34db4935d5f815a8c687d0e2741123afd1ce7e7bbac9dc96f2" Oct 30 01:15:03.464629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.464602 4905 generic.go:356] "Generic (PLEG): container finished" podID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerID="b1349aab26591b0d493a9111fc6e17676e8e1cd459d67da46e42bcf552a5a763" exitCode=60 Oct 30 01:15:03.465146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.465130 4905 scope.go:122] "RemoveContainer" containerID="b1349aab26591b0d493a9111fc6e17676e8e1cd459d67da46e42bcf552a5a763" Oct 30 01:15:03.465769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.465660 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" event={"ID":"9a049cef-bb00-46d6-85e9-7fbd280b3d0e","Type":"ContainerDied","Data":"b1349aab26591b0d493a9111fc6e17676e8e1cd459d67da46e42bcf552a5a763"} Oct 30 01:15:03.467422 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.467334 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" event={"ID":"b168dc55-dfba-45d0-b281-80390072df92","Type":"ContainerStarted","Data":"16a8afce227d6c0566d5c36b0891432b2d7559e8785d825ea7ebc5821216ecc3"} Oct 30 01:15:03.469360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.469335 4905 generic.go:356] "Generic (PLEG): container finished" podID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerID="f9d8cc66e2e7c664715affbe6b2fa8d6e2d8a9bf61454d736464470ee207900a" exitCode=60 Oct 30 01:15:03.469462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.469441 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" event={"ID":"31e0c08a-049e-484e-aab6-ddeeed75c2f1","Type":"ContainerDied","Data":"f9d8cc66e2e7c664715affbe6b2fa8d6e2d8a9bf61454d736464470ee207900a"} Oct 30 01:15:03.469680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.469662 4905 scope.go:122] "RemoveContainer" containerID="f9d8cc66e2e7c664715affbe6b2fa8d6e2d8a9bf61454d736464470ee207900a" Oct 30 01:15:03.471472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.471449 4905 generic.go:356] "Generic (PLEG): container finished" podID="f379fc6d-07de-491f-8c4b-210a84498a91" containerID="cf35f41d3fc2c687ebbc66c7a9f571bc239c880a1c330d4b6e96585c35f447e4" exitCode=7 Oct 30 01:15:03.471588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.471501 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" event={"ID":"f379fc6d-07de-491f-8c4b-210a84498a91","Type":"ContainerDied","Data":"cf35f41d3fc2c687ebbc66c7a9f571bc239c880a1c330d4b6e96585c35f447e4"} Oct 30 01:15:03.471901 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.471882 4905 scope.go:122] "RemoveContainer" containerID="cf35f41d3fc2c687ebbc66c7a9f571bc239c880a1c330d4b6e96585c35f447e4" Oct 30 01:15:03.620400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:03.620349 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" podStartSLOduration=3.620336996 podStartE2EDuration="3.620336996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.618590951 +0000 UTC m=+2188.561764143" watchObservedRunningTime="2026-10-30 01:15:03.620336996 +0000 UTC m=+2188.563510188" Oct 30 01:15:04.485588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.485557 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerStarted","Data":"441ae06f9c4a5f65eb22878516adf101a81aedb8da795f7c287bb104ab2fb880"} Oct 30 01:15:04.489150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.489120 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" event={"ID":"54a4faf7-b736-44ef-bc82-195af1bf0af8","Type":"ContainerStarted","Data":"460f5461a81e4625df7b0746236e42fbc37fcf3257758f222ac57bce51a31c77"} Oct 30 01:15:04.491651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.491581 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" event={"ID":"9a049cef-bb00-46d6-85e9-7fbd280b3d0e","Type":"ContainerStarted","Data":"55fbd97c063191c875d8e1e0dcba5aeb2c554c3896f477218bc380415b038457"} Oct 30 01:15:04.493666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.493641 4905 generic.go:356] "Generic (PLEG): container finished" podID="b168dc55-dfba-45d0-b281-80390072df92" containerID="16a8afce227d6c0566d5c36b0891432b2d7559e8785d825ea7ebc5821216ecc3" exitCode=60 Oct 30 01:15:04.493760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.493689 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" event={"ID":"b168dc55-dfba-45d0-b281-80390072df92","Type":"ContainerDied","Data":"16a8afce227d6c0566d5c36b0891432b2d7559e8785d825ea7ebc5821216ecc3"} Oct 30 01:15:04.493760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.493736 4905 scope.go:122] "RemoveContainer" containerID="5a5f7b61059ce5cc58d7ee2fdccecda7650d65be07a0bb9eb26630f437799ee6" Oct 30 01:15:04.494008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.493992 4905 scope.go:122] "RemoveContainer" containerID="16a8afce227d6c0566d5c36b0891432b2d7559e8785d825ea7ebc5821216ecc3" Oct 30 01:15:04.494368 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:04.494335 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl_odh-ai-gateway-infra(b168dc55-dfba-45d0-b281-80390072df92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" podUID="b168dc55-dfba-45d0-b281-80390072df92" Oct 30 01:15:04.495875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.495854 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" event={"ID":"31e0c08a-049e-484e-aab6-ddeeed75c2f1","Type":"ContainerStarted","Data":"e32533aebbb2f44bd22f6d9b3e997e6ee2a9c350e953423e8d095556b6211bdf"} Oct 30 01:15:04.497948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.497927 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" event={"ID":"f379fc6d-07de-491f-8c4b-210a84498a91","Type":"ContainerStarted","Data":"1613e092d395de2300e3326a198db23d6b091ee75e2c9b56f61bad188e23cd55"} Oct 30 01:15:04.500464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.500437 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" event={"ID":"33374e9d-1513-4645-a044-d979e11ac979","Type":"ContainerStarted","Data":"4ee07288edc472726863d3d386936957a4b8ce56bac889f771a69e3b82429701"} Oct 30 01:15:04.502709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.502686 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" event={"ID":"c40fd573-8d64-425e-9134-054f7440e187","Type":"ContainerStarted","Data":"94e532f651c76920977be70de2593cfaec237a8bf8eae55f79b5135a1a3119ff"} Oct 30 01:15:04.504493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:04.504474 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" event={"ID":"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56","Type":"ContainerStarted","Data":"d001a024f4747318266cad697f9b05501b96944fc58216d68fc9a587a41b544f"} Oct 30 01:15:05.514394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.514356 4905 generic.go:356] "Generic (PLEG): container finished" podID="33374e9d-1513-4645-a044-d979e11ac979" containerID="4ee07288edc472726863d3d386936957a4b8ce56bac889f771a69e3b82429701" exitCode=60 Oct 30 01:15:05.514918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.514449 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" event={"ID":"33374e9d-1513-4645-a044-d979e11ac979","Type":"ContainerDied","Data":"4ee07288edc472726863d3d386936957a4b8ce56bac889f771a69e3b82429701"} Oct 30 01:15:05.514918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.514599 4905 scope.go:122] "RemoveContainer" containerID="cf4ab4f8a7d942c32fcdee5110c9198ebf752809a1d163533e137fa0650b4852" Oct 30 01:15:05.514918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.514753 4905 scope.go:122] "RemoveContainer" containerID="4ee07288edc472726863d3d386936957a4b8ce56bac889f771a69e3b82429701" Oct 30 01:15:05.515162 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.515135 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl_odh-ai-gateway-infra(33374e9d-1513-4645-a044-d979e11ac979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" podUID="33374e9d-1513-4645-a044-d979e11ac979" Oct 30 01:15:05.516829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.516808 4905 generic.go:356] "Generic (PLEG): container finished" podID="c40fd573-8d64-425e-9134-054f7440e187" containerID="94e532f651c76920977be70de2593cfaec237a8bf8eae55f79b5135a1a3119ff" exitCode=60 Oct 30 01:15:05.516967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.516868 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" event={"ID":"c40fd573-8d64-425e-9134-054f7440e187","Type":"ContainerDied","Data":"94e532f651c76920977be70de2593cfaec237a8bf8eae55f79b5135a1a3119ff"} Oct 30 01:15:05.517145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.517131 4905 scope.go:122] "RemoveContainer" containerID="94e532f651c76920977be70de2593cfaec237a8bf8eae55f79b5135a1a3119ff" Oct 30 01:15:05.517576 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.517473 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg_odh-ai-gateway-infra(c40fd573-8d64-425e-9134-054f7440e187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" podUID="c40fd573-8d64-425e-9134-054f7440e187" Oct 30 01:15:05.519108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.519085 4905 generic.go:356] "Generic (PLEG): container finished" podID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerID="d001a024f4747318266cad697f9b05501b96944fc58216d68fc9a587a41b544f" exitCode=60 Oct 30 01:15:05.519207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.519168 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" event={"ID":"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56","Type":"ContainerDied","Data":"d001a024f4747318266cad697f9b05501b96944fc58216d68fc9a587a41b544f"} Oct 30 01:15:05.519444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.519428 4905 scope.go:122] "RemoveContainer" containerID="d001a024f4747318266cad697f9b05501b96944fc58216d68fc9a587a41b544f" Oct 30 01:15:05.519752 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.519734 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf_odh-ai-gateway-infra(3b4b96d8-ffc3-4e55-83de-4a3b8d216b56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" Oct 30 01:15:05.521563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.521540 4905 generic.go:356] "Generic (PLEG): container finished" podID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerID="441ae06f9c4a5f65eb22878516adf101a81aedb8da795f7c287bb104ab2fb880" exitCode=60 Oct 30 01:15:05.521657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.521601 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerDied","Data":"441ae06f9c4a5f65eb22878516adf101a81aedb8da795f7c287bb104ab2fb880"} Oct 30 01:15:05.521837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.521824 4905 scope.go:122] "RemoveContainer" containerID="441ae06f9c4a5f65eb22878516adf101a81aedb8da795f7c287bb104ab2fb880" Oct 30 01:15:05.522105 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.522085 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8_odh-ai-gateway-infra(f1ff5453-b37c-496b-8abc-e95ff86338df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" Oct 30 01:15:05.523632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.523608 4905 generic.go:356] "Generic (PLEG): container finished" podID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerID="460f5461a81e4625df7b0746236e42fbc37fcf3257758f222ac57bce51a31c77" exitCode=60 Oct 30 01:15:05.523710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.523694 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" event={"ID":"54a4faf7-b736-44ef-bc82-195af1bf0af8","Type":"ContainerDied","Data":"460f5461a81e4625df7b0746236e42fbc37fcf3257758f222ac57bce51a31c77"} Oct 30 01:15:05.523933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.523911 4905 scope.go:122] "RemoveContainer" containerID="460f5461a81e4625df7b0746236e42fbc37fcf3257758f222ac57bce51a31c77" Oct 30 01:15:05.524235 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.524208 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg_odh-ai-gateway-infra(54a4faf7-b736-44ef-bc82-195af1bf0af8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" Oct 30 01:15:05.525762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.525740 4905 generic.go:356] "Generic (PLEG): container finished" podID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerID="55fbd97c063191c875d8e1e0dcba5aeb2c554c3896f477218bc380415b038457" exitCode=60 Oct 30 01:15:05.525879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.525796 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" event={"ID":"9a049cef-bb00-46d6-85e9-7fbd280b3d0e","Type":"ContainerDied","Data":"55fbd97c063191c875d8e1e0dcba5aeb2c554c3896f477218bc380415b038457"} Oct 30 01:15:05.526052 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.526036 4905 scope.go:122] "RemoveContainer" containerID="55fbd97c063191c875d8e1e0dcba5aeb2c554c3896f477218bc380415b038457" Oct 30 01:15:05.526274 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.526254 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-lr27c_odh-ai-gateway-infra(9a049cef-bb00-46d6-85e9-7fbd280b3d0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" Oct 30 01:15:05.527706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.527690 4905 scope.go:122] "RemoveContainer" containerID="16a8afce227d6c0566d5c36b0891432b2d7559e8785d825ea7ebc5821216ecc3" Oct 30 01:15:05.527951 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.527925 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl_odh-ai-gateway-infra(b168dc55-dfba-45d0-b281-80390072df92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" podUID="b168dc55-dfba-45d0-b281-80390072df92" Oct 30 01:15:05.529557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.529509 4905 generic.go:356] "Generic (PLEG): container finished" podID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerID="e32533aebbb2f44bd22f6d9b3e997e6ee2a9c350e953423e8d095556b6211bdf" exitCode=60 Oct 30 01:15:05.529665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.529620 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" event={"ID":"31e0c08a-049e-484e-aab6-ddeeed75c2f1","Type":"ContainerDied","Data":"e32533aebbb2f44bd22f6d9b3e997e6ee2a9c350e953423e8d095556b6211bdf"} Oct 30 01:15:05.529844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.529822 4905 scope.go:122] "RemoveContainer" containerID="e32533aebbb2f44bd22f6d9b3e997e6ee2a9c350e953423e8d095556b6211bdf" Oct 30 01:15:05.530140 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.530089 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k_odh-ai-gateway-infra(31e0c08a-049e-484e-aab6-ddeeed75c2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" Oct 30 01:15:05.532661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.532445 4905 scope.go:122] "RemoveContainer" containerID="0123daeeb57c9e2b0a2408909edeb3044e451a23150974313807850442fd7da3" Oct 30 01:15:05.534958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.534937 4905 generic.go:356] "Generic (PLEG): container finished" podID="f379fc6d-07de-491f-8c4b-210a84498a91" containerID="1613e092d395de2300e3326a198db23d6b091ee75e2c9b56f61bad188e23cd55" exitCode=7 Oct 30 01:15:05.535163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.535115 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" event={"ID":"f379fc6d-07de-491f-8c4b-210a84498a91","Type":"ContainerDied","Data":"1613e092d395de2300e3326a198db23d6b091ee75e2c9b56f61bad188e23cd55"} Oct 30 01:15:05.535334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.535318 4905 scope.go:122] "RemoveContainer" containerID="1613e092d395de2300e3326a198db23d6b091ee75e2c9b56f61bad188e23cd55" Oct 30 01:15:05.535721 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:05.535695 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj_odh-ai-gateway-infra(f379fc6d-07de-491f-8c4b-210a84498a91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" Oct 30 01:15:05.547976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.547960 4905 scope.go:122] "RemoveContainer" containerID="db6180ca63a7120426804b9d1c5e74a6a928bfb53c02cbbc05b9ef6411e73d20" Oct 30 01:15:05.561368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.561348 4905 scope.go:122] "RemoveContainer" containerID="c8d3a879223893ce1332fdcff7e2b000ff0fddd6fc69d02a4b258f1102c390a8" Oct 30 01:15:05.574893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.574877 4905 scope.go:122] "RemoveContainer" containerID="85dc04caba2a9b34db4935d5f815a8c687d0e2741123afd1ce7e7bbac9dc96f2" Oct 30 01:15:05.588502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.588484 4905 scope.go:122] "RemoveContainer" containerID="b1349aab26591b0d493a9111fc6e17676e8e1cd459d67da46e42bcf552a5a763" Oct 30 01:15:05.602454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.602431 4905 scope.go:122] "RemoveContainer" containerID="f9d8cc66e2e7c664715affbe6b2fa8d6e2d8a9bf61454d736464470ee207900a" Oct 30 01:15:05.615623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:05.615606 4905 scope.go:122] "RemoveContainer" containerID="cf35f41d3fc2c687ebbc66c7a9f571bc239c880a1c330d4b6e96585c35f447e4" Oct 30 01:15:06.544743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.544714 4905 scope.go:122] "RemoveContainer" containerID="94e532f651c76920977be70de2593cfaec237a8bf8eae55f79b5135a1a3119ff" Oct 30 01:15:06.545209 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.545079 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg_odh-ai-gateway-infra(c40fd573-8d64-425e-9134-054f7440e187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" podUID="c40fd573-8d64-425e-9134-054f7440e187" Oct 30 01:15:06.546706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.546680 4905 scope.go:122] "RemoveContainer" containerID="d001a024f4747318266cad697f9b05501b96944fc58216d68fc9a587a41b544f" Oct 30 01:15:06.546987 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.546958 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf_odh-ai-gateway-infra(3b4b96d8-ffc3-4e55-83de-4a3b8d216b56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" Oct 30 01:15:06.548656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.548634 4905 scope.go:122] "RemoveContainer" containerID="441ae06f9c4a5f65eb22878516adf101a81aedb8da795f7c287bb104ab2fb880" Oct 30 01:15:06.548930 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.548892 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8_odh-ai-gateway-infra(f1ff5453-b37c-496b-8abc-e95ff86338df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" Oct 30 01:15:06.550343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.550327 4905 scope.go:122] "RemoveContainer" containerID="460f5461a81e4625df7b0746236e42fbc37fcf3257758f222ac57bce51a31c77" Oct 30 01:15:06.550626 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.550599 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg_odh-ai-gateway-infra(54a4faf7-b736-44ef-bc82-195af1bf0af8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" Oct 30 01:15:06.551799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.551782 4905 scope.go:122] "RemoveContainer" containerID="55fbd97c063191c875d8e1e0dcba5aeb2c554c3896f477218bc380415b038457" Oct 30 01:15:06.552030 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.552013 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-lr27c_odh-ai-gateway-infra(9a049cef-bb00-46d6-85e9-7fbd280b3d0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" Oct 30 01:15:06.553535 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.553503 4905 scope.go:122] "RemoveContainer" containerID="e32533aebbb2f44bd22f6d9b3e997e6ee2a9c350e953423e8d095556b6211bdf" Oct 30 01:15:06.553770 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.553746 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k_odh-ai-gateway-infra(31e0c08a-049e-484e-aab6-ddeeed75c2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" Oct 30 01:15:06.555309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.555293 4905 scope.go:122] "RemoveContainer" containerID="1613e092d395de2300e3326a198db23d6b091ee75e2c9b56f61bad188e23cd55" Oct 30 01:15:06.555565 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.555507 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj_odh-ai-gateway-infra(f379fc6d-07de-491f-8c4b-210a84498a91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" Oct 30 01:15:06.556899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:06.556884 4905 scope.go:122] "RemoveContainer" containerID="4ee07288edc472726863d3d386936957a4b8ce56bac889f771a69e3b82429701" Oct 30 01:15:06.557068 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:06.557053 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl_odh-ai-gateway-infra(33374e9d-1513-4645-a044-d979e11ac979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" podUID="33374e9d-1513-4645-a044-d979e11ac979" Oct 30 01:15:12.416048 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:12.416007 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:15:14.182965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:14.182938 4905 scope.go:122] "RemoveContainer" containerID="16a8afce227d6c0566d5c36b0891432b2d7559e8785d825ea7ebc5821216ecc3" Oct 30 01:15:15.187613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:15.187586 4905 scope.go:122] "RemoveContainer" containerID="94e532f651c76920977be70de2593cfaec237a8bf8eae55f79b5135a1a3119ff" Oct 30 01:15:15.188015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:15.187621 4905 scope.go:122] "RemoveContainer" containerID="d001a024f4747318266cad697f9b05501b96944fc58216d68fc9a587a41b544f" Oct 30 01:15:15.636962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:15.636934 4905 generic.go:356] "Generic (PLEG): container finished" podID="b168dc55-dfba-45d0-b281-80390072df92" containerID="f2b7f4cfa2f313e11e2b379e97eebb4de7e5ceed87491db9031c451e71367630" exitCode=60 Oct 30 01:15:15.637084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:15.637010 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" event={"ID":"b168dc55-dfba-45d0-b281-80390072df92","Type":"ContainerDied","Data":"f2b7f4cfa2f313e11e2b379e97eebb4de7e5ceed87491db9031c451e71367630"} Oct 30 01:15:15.637313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:15.637086 4905 scope.go:122] "RemoveContainer" containerID="16a8afce227d6c0566d5c36b0891432b2d7559e8785d825ea7ebc5821216ecc3" Oct 30 01:15:15.637712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:15.637674 4905 scope.go:122] "RemoveContainer" containerID="f2b7f4cfa2f313e11e2b379e97eebb4de7e5ceed87491db9031c451e71367630" Oct 30 01:15:15.638103 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:15.638078 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl_odh-ai-gateway-infra(b168dc55-dfba-45d0-b281-80390072df92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" podUID="b168dc55-dfba-45d0-b281-80390072df92" Oct 30 01:15:16.183909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.183885 4905 scope.go:122] "RemoveContainer" containerID="4ee07288edc472726863d3d386936957a4b8ce56bac889f771a69e3b82429701" Oct 30 01:15:16.184068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.184006 4905 scope.go:122] "RemoveContainer" containerID="e32533aebbb2f44bd22f6d9b3e997e6ee2a9c350e953423e8d095556b6211bdf" Oct 30 01:15:16.184107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.184081 4905 scope.go:122] "RemoveContainer" containerID="1613e092d395de2300e3326a198db23d6b091ee75e2c9b56f61bad188e23cd55" Oct 30 01:15:16.648794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.648759 4905 generic.go:356] "Generic (PLEG): container finished" podID="c40fd573-8d64-425e-9134-054f7440e187" containerID="b784ecc6b5366254256fa36338e8e0eabd202bef9ed3efe3194b5bccb8c6de8f" exitCode=60 Oct 30 01:15:16.649215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.648794 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" event={"ID":"c40fd573-8d64-425e-9134-054f7440e187","Type":"ContainerDied","Data":"b784ecc6b5366254256fa36338e8e0eabd202bef9ed3efe3194b5bccb8c6de8f"} Oct 30 01:15:16.649215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.648878 4905 scope.go:122] "RemoveContainer" containerID="94e532f651c76920977be70de2593cfaec237a8bf8eae55f79b5135a1a3119ff" Oct 30 01:15:16.649457 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.649435 4905 scope.go:122] "RemoveContainer" containerID="b784ecc6b5366254256fa36338e8e0eabd202bef9ed3efe3194b5bccb8c6de8f" Oct 30 01:15:16.649817 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:16.649787 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg_odh-ai-gateway-infra(c40fd573-8d64-425e-9134-054f7440e187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" podUID="c40fd573-8d64-425e-9134-054f7440e187" Oct 30 01:15:16.652478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.652456 4905 generic.go:356] "Generic (PLEG): container finished" podID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerID="17b67d1436eaaf4bf42de0de5f21d5abbea73b625b856beae1bdfb73425027ea" exitCode=60 Oct 30 01:15:16.652695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.652654 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" event={"ID":"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56","Type":"ContainerDied","Data":"17b67d1436eaaf4bf42de0de5f21d5abbea73b625b856beae1bdfb73425027ea"} Oct 30 01:15:16.653038 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.653002 4905 scope.go:122] "RemoveContainer" containerID="17b67d1436eaaf4bf42de0de5f21d5abbea73b625b856beae1bdfb73425027ea" Oct 30 01:15:16.653409 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:16.653380 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf_odh-ai-gateway-infra(3b4b96d8-ffc3-4e55-83de-4a3b8d216b56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" Oct 30 01:15:16.665990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.665957 4905 scope.go:122] "RemoveContainer" containerID="d001a024f4747318266cad697f9b05501b96944fc58216d68fc9a587a41b544f" Oct 30 01:15:16.694478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.694429 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl"] Oct 30 01:15:16.959642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:16.959573 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:17.059043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.058979 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b168dc55-dfba-45d0-b281-80390072df92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b168dc55-dfba-45d0-b281-80390072df92-maas-api-service-ca\") pod \"b168dc55-dfba-45d0-b281-80390072df92\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " Oct 30 01:15:17.060201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.060178 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b168dc55-dfba-45d0-b281-80390072df92-kube-api-access-dmbdq\" (UniqueName: \"kubernetes.io/projected/b168dc55-dfba-45d0-b281-80390072df92-kube-api-access-dmbdq\") pod \"b168dc55-dfba-45d0-b281-80390072df92\" (UID: \"b168dc55-dfba-45d0-b281-80390072df92\") " Oct 30 01:15:17.060461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.060067 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b168dc55-dfba-45d0-b281-80390072df92-maas-api-service-ca" pod "b168dc55-dfba-45d0-b281-80390072df92" (UID: "b168dc55-dfba-45d0-b281-80390072df92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:17.061631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.061504 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b168dc55-dfba-45d0-b281-80390072df92-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:17.067871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.067845 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b168dc55-dfba-45d0-b281-80390072df92-kube-api-access-dmbdq" pod "b168dc55-dfba-45d0-b281-80390072df92" (UID: "b168dc55-dfba-45d0-b281-80390072df92"). InnerVolumeSpecName "kube-api-access-dmbdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:17.163827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.163797 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmbdq\" (UniqueName: \"kubernetes.io/projected/b168dc55-dfba-45d0-b281-80390072df92-kube-api-access-dmbdq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:17.195914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.195891 4905 scope.go:122] "RemoveContainer" containerID="460f5461a81e4625df7b0746236e42fbc37fcf3257758f222ac57bce51a31c77" Oct 30 01:15:17.690339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.690310 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" Oct 30 01:15:17.690856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.690331 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl" event={"ID":"b168dc55-dfba-45d0-b281-80390072df92","Type":"ContainerDied","Data":"98daacdb0af8f81b3641e68b77986fbb16f8cf45a3ce2360e0abb468d628957f"} Oct 30 01:15:17.690856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.690428 4905 scope.go:122] "RemoveContainer" containerID="f2b7f4cfa2f313e11e2b379e97eebb4de7e5ceed87491db9031c451e71367630" Oct 30 01:15:17.693893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.693868 4905 generic.go:356] "Generic (PLEG): container finished" podID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerID="2fa9e6d1929fe63de2321250d7d8c77d1d8b50b7373849a721ece1b1a1e192b8" exitCode=60 Oct 30 01:15:17.693986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.693935 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" event={"ID":"31e0c08a-049e-484e-aab6-ddeeed75c2f1","Type":"ContainerDied","Data":"2fa9e6d1929fe63de2321250d7d8c77d1d8b50b7373849a721ece1b1a1e192b8"} Oct 30 01:15:17.694278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.694259 4905 scope.go:122] "RemoveContainer" containerID="2fa9e6d1929fe63de2321250d7d8c77d1d8b50b7373849a721ece1b1a1e192b8" Oct 30 01:15:17.694583 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:17.694559 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k_odh-ai-gateway-infra(31e0c08a-049e-484e-aab6-ddeeed75c2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" Oct 30 01:15:17.696169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.696149 4905 generic.go:356] "Generic (PLEG): container finished" podID="f379fc6d-07de-491f-8c4b-210a84498a91" containerID="078c2f9204061006b0b05e5540de65953279a23499a779568e3ce9f78e93cde1" exitCode=60 Oct 30 01:15:17.696283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.696218 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" event={"ID":"f379fc6d-07de-491f-8c4b-210a84498a91","Type":"ContainerDied","Data":"078c2f9204061006b0b05e5540de65953279a23499a779568e3ce9f78e93cde1"} Oct 30 01:15:17.696612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.696510 4905 scope.go:122] "RemoveContainer" containerID="078c2f9204061006b0b05e5540de65953279a23499a779568e3ce9f78e93cde1" Oct 30 01:15:17.696815 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:17.696793 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj_odh-ai-gateway-infra(f379fc6d-07de-491f-8c4b-210a84498a91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" Oct 30 01:15:17.698290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.698270 4905 generic.go:356] "Generic (PLEG): container finished" podID="33374e9d-1513-4645-a044-d979e11ac979" containerID="6ce8d5bbf1af9958a426128e8c2508bc035584251a5b5e51015754bfd66b8e2d" exitCode=60 Oct 30 01:15:17.698390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.698337 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" event={"ID":"33374e9d-1513-4645-a044-d979e11ac979","Type":"ContainerDied","Data":"6ce8d5bbf1af9958a426128e8c2508bc035584251a5b5e51015754bfd66b8e2d"} Oct 30 01:15:17.698682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.698666 4905 scope.go:122] "RemoveContainer" containerID="6ce8d5bbf1af9958a426128e8c2508bc035584251a5b5e51015754bfd66b8e2d" Oct 30 01:15:17.698908 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:17.698888 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl_odh-ai-gateway-infra(33374e9d-1513-4645-a044-d979e11ac979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" podUID="33374e9d-1513-4645-a044-d979e11ac979" Oct 30 01:15:17.703566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.703550 4905 scope.go:122] "RemoveContainer" containerID="e32533aebbb2f44bd22f6d9b3e997e6ee2a9c350e953423e8d095556b6211bdf" Oct 30 01:15:17.706557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.706503 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg"] Oct 30 01:15:17.721783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.721759 4905 scope.go:122] "RemoveContainer" containerID="1613e092d395de2300e3326a198db23d6b091ee75e2c9b56f61bad188e23cd55" Oct 30 01:15:17.738916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.738898 4905 scope.go:122] "RemoveContainer" containerID="4ee07288edc472726863d3d386936957a4b8ce56bac889f771a69e3b82429701" Oct 30 01:15:17.746297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.746276 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf"] Oct 30 01:15:17.890925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.890481 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl"] Oct 30 01:15:17.898277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.898241 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b05110-29888715-x89pl"] Oct 30 01:15:17.989986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:17.989956 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:18.028431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.028393 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:18.131555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.130333 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-kube-api-access-2wgr2\" (UniqueName: \"kubernetes.io/projected/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-kube-api-access-2wgr2\") pod \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " Oct 30 01:15:18.131555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.130512 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40fd573-8d64-425e-9134-054f7440e187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40fd573-8d64-425e-9134-054f7440e187-maas-api-service-ca\") pod \"c40fd573-8d64-425e-9134-054f7440e187\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " Oct 30 01:15:18.131555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.130634 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-maas-api-service-ca\") pod \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\" (UID: \"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56\") " Oct 30 01:15:18.131555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.130775 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40fd573-8d64-425e-9134-054f7440e187-kube-api-access-47grb\" (UniqueName: \"kubernetes.io/projected/c40fd573-8d64-425e-9134-054f7440e187-kube-api-access-47grb\") pod \"c40fd573-8d64-425e-9134-054f7440e187\" (UID: \"c40fd573-8d64-425e-9134-054f7440e187\") " Oct 30 01:15:18.167552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.166928 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-maas-api-service-ca" pod "3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" (UID: "3b4b96d8-ffc3-4e55-83de-4a3b8d216b56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:18.167552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.167072 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40fd573-8d64-425e-9134-054f7440e187-maas-api-service-ca" pod "c40fd573-8d64-425e-9134-054f7440e187" (UID: "c40fd573-8d64-425e-9134-054f7440e187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:18.178386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.176405 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40fd573-8d64-425e-9134-054f7440e187-kube-api-access-47grb" pod "c40fd573-8d64-425e-9134-054f7440e187" (UID: "c40fd573-8d64-425e-9134-054f7440e187"). InnerVolumeSpecName "kube-api-access-47grb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:18.180781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.180744 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-kube-api-access-2wgr2" pod "3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" (UID: "3b4b96d8-ffc3-4e55-83de-4a3b8d216b56"). InnerVolumeSpecName "kube-api-access-2wgr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:18.183056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.183030 4905 scope.go:122] "RemoveContainer" containerID="441ae06f9c4a5f65eb22878516adf101a81aedb8da795f7c287bb104ab2fb880" Oct 30 01:15:18.195615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.195430 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47grb\" (UniqueName: \"kubernetes.io/projected/c40fd573-8d64-425e-9134-054f7440e187-kube-api-access-47grb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:18.195615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.195458 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wgr2\" (UniqueName: \"kubernetes.io/projected/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-kube-api-access-2wgr2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:18.195615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.195474 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40fd573-8d64-425e-9134-054f7440e187-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:18.195615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.195488 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:18.717214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.717129 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" Oct 30 01:15:18.717214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.717145 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg" event={"ID":"c40fd573-8d64-425e-9134-054f7440e187","Type":"ContainerDied","Data":"560d266bf2b2a2a70cbcd704c019f8cfaac1a03d34c600d0efe5cfbe9401aafa"} Oct 30 01:15:18.717214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.717204 4905 scope.go:122] "RemoveContainer" containerID="b784ecc6b5366254256fa36338e8e0eabd202bef9ed3efe3194b5bccb8c6de8f" Oct 30 01:15:18.719359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.719341 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" Oct 30 01:15:18.719359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.719347 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf" event={"ID":"3b4b96d8-ffc3-4e55-83de-4a3b8d216b56","Type":"ContainerDied","Data":"574a27cedabd82dc14c783ee2ce202b01e7ea5182cd59d10b416d1c1080838f7"} Oct 30 01:15:18.721507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.721464 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerStarted","Data":"d073d4511db0eac0be783ca9cedf80f27f48342e67256ec390f02998d32a7837"} Oct 30 01:15:18.728852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.727393 4905 generic.go:356] "Generic (PLEG): container finished" podID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerID="cafc098171720047ade7df6efd6b49b1e9cfa71c24668035fc05c8c8c7bd02a4" exitCode=60 Oct 30 01:15:18.728852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.727553 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" event={"ID":"54a4faf7-b736-44ef-bc82-195af1bf0af8","Type":"ContainerDied","Data":"cafc098171720047ade7df6efd6b49b1e9cfa71c24668035fc05c8c8c7bd02a4"} Oct 30 01:15:18.728852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.727997 4905 scope.go:122] "RemoveContainer" containerID="cafc098171720047ade7df6efd6b49b1e9cfa71c24668035fc05c8c8c7bd02a4" Oct 30 01:15:18.728852 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:18.728437 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg_odh-ai-gateway-infra(54a4faf7-b736-44ef-bc82-195af1bf0af8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" Oct 30 01:15:18.769029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.768999 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k"] Oct 30 01:15:18.773452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.773426 4905 scope.go:122] "RemoveContainer" containerID="17b67d1436eaaf4bf42de0de5f21d5abbea73b625b856beae1bdfb73425027ea" Oct 30 01:15:18.792698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.792670 4905 scope.go:122] "RemoveContainer" containerID="460f5461a81e4625df7b0746236e42fbc37fcf3257758f222ac57bce51a31c77" Oct 30 01:15:18.851737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.851700 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj"] Oct 30 01:15:18.887639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.885387 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg"] Oct 30 01:15:18.898300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.895247 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-283eb6-29888715-jn9hg"] Oct 30 01:15:18.924050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.924020 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf"] Oct 30 01:15:18.959178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.959147 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b565ec-29888715-w67gf"] Oct 30 01:15:18.964136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:18.964108 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl"] Oct 30 01:15:19.032241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.032180 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:19.089505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.089475 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:19.112742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.112706 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e0c08a-049e-484e-aab6-ddeeed75c2f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e0c08a-049e-484e-aab6-ddeeed75c2f1-maas-api-service-ca\") pod \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " Oct 30 01:15:19.112906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.112782 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f379fc6d-07de-491f-8c4b-210a84498a91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f379fc6d-07de-491f-8c4b-210a84498a91-maas-api-service-ca\") pod \"f379fc6d-07de-491f-8c4b-210a84498a91\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " Oct 30 01:15:19.112906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.112821 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f379fc6d-07de-491f-8c4b-210a84498a91-kube-api-access-q4vpx\" (UniqueName: \"kubernetes.io/projected/f379fc6d-07de-491f-8c4b-210a84498a91-kube-api-access-q4vpx\") pod \"f379fc6d-07de-491f-8c4b-210a84498a91\" (UID: \"f379fc6d-07de-491f-8c4b-210a84498a91\") " Oct 30 01:15:19.113104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.112972 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e0c08a-049e-484e-aab6-ddeeed75c2f1-kube-api-access-cqzkg\" (UniqueName: \"kubernetes.io/projected/31e0c08a-049e-484e-aab6-ddeeed75c2f1-kube-api-access-cqzkg\") pod \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\" (UID: \"31e0c08a-049e-484e-aab6-ddeeed75c2f1\") " Oct 30 01:15:19.113455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.113429 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f379fc6d-07de-491f-8c4b-210a84498a91-maas-api-service-ca" pod "f379fc6d-07de-491f-8c4b-210a84498a91" (UID: "f379fc6d-07de-491f-8c4b-210a84498a91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.114079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.113925 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f379fc6d-07de-491f-8c4b-210a84498a91-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.114079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.113964 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e0c08a-049e-484e-aab6-ddeeed75c2f1-maas-api-service-ca" pod "31e0c08a-049e-484e-aab6-ddeeed75c2f1" (UID: "31e0c08a-049e-484e-aab6-ddeeed75c2f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.116749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.116689 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f379fc6d-07de-491f-8c4b-210a84498a91-kube-api-access-q4vpx" pod "f379fc6d-07de-491f-8c4b-210a84498a91" (UID: "f379fc6d-07de-491f-8c4b-210a84498a91"). InnerVolumeSpecName "kube-api-access-q4vpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.117322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.117297 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e0c08a-049e-484e-aab6-ddeeed75c2f1-kube-api-access-cqzkg" pod "31e0c08a-049e-484e-aab6-ddeeed75c2f1" (UID: "31e0c08a-049e-484e-aab6-ddeeed75c2f1"). InnerVolumeSpecName "kube-api-access-cqzkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.195387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.191147 4905 scope.go:122] "RemoveContainer" containerID="55fbd97c063191c875d8e1e0dcba5aeb2c554c3896f477218bc380415b038457" Oct 30 01:15:19.295761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.295171 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vpx\" (UniqueName: \"kubernetes.io/projected/f379fc6d-07de-491f-8c4b-210a84498a91-kube-api-access-q4vpx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.295761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.295200 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqzkg\" (UniqueName: \"kubernetes.io/projected/31e0c08a-049e-484e-aab6-ddeeed75c2f1-kube-api-access-cqzkg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.295761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.295215 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e0c08a-049e-484e-aab6-ddeeed75c2f1-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.322879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.322228 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" path="/var/lib/kubelet/pods/3b4b96d8-ffc3-4e55-83de-4a3b8d216b56/volumes" Oct 30 01:15:19.323627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.323603 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b168dc55-dfba-45d0-b281-80390072df92" path="/var/lib/kubelet/pods/b168dc55-dfba-45d0-b281-80390072df92/volumes" Oct 30 01:15:19.324442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.324420 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40fd573-8d64-425e-9134-054f7440e187" path="/var/lib/kubelet/pods/c40fd573-8d64-425e-9134-054f7440e187/volumes" Oct 30 01:15:19.410341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.410316 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:19.498041 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.497981 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33374e9d-1513-4645-a044-d979e11ac979-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33374e9d-1513-4645-a044-d979e11ac979-maas-api-service-ca\") pod \"33374e9d-1513-4645-a044-d979e11ac979\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " Oct 30 01:15:19.498890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.498824 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33374e9d-1513-4645-a044-d979e11ac979-kube-api-access-ct2p6\" (UniqueName: \"kubernetes.io/projected/33374e9d-1513-4645-a044-d979e11ac979-kube-api-access-ct2p6\") pod \"33374e9d-1513-4645-a044-d979e11ac979\" (UID: \"33374e9d-1513-4645-a044-d979e11ac979\") " Oct 30 01:15:19.528595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.498754 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33374e9d-1513-4645-a044-d979e11ac979-maas-api-service-ca" pod "33374e9d-1513-4645-a044-d979e11ac979" (UID: "33374e9d-1513-4645-a044-d979e11ac979"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.569188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.569153 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33374e9d-1513-4645-a044-d979e11ac979-kube-api-access-ct2p6" pod "33374e9d-1513-4645-a044-d979e11ac979" (UID: "33374e9d-1513-4645-a044-d979e11ac979"). InnerVolumeSpecName "kube-api-access-ct2p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.633985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.633952 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33374e9d-1513-4645-a044-d979e11ac979-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.633985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.633985 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct2p6\" (UniqueName: \"kubernetes.io/projected/33374e9d-1513-4645-a044-d979e11ac979-kube-api-access-ct2p6\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.753025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.752993 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" Oct 30 01:15:19.753494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.753037 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k" event={"ID":"31e0c08a-049e-484e-aab6-ddeeed75c2f1","Type":"ContainerDied","Data":"11552f7aa7337a896c5271e6e93fa248593650d056f873afdbb9fe716dcc365a"} Oct 30 01:15:19.753494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.753112 4905 scope.go:122] "RemoveContainer" containerID="2fa9e6d1929fe63de2321250d7d8c77d1d8b50b7373849a721ece1b1a1e192b8" Oct 30 01:15:19.758868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.758845 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" Oct 30 01:15:19.759185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.758852 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj" event={"ID":"f379fc6d-07de-491f-8c4b-210a84498a91","Type":"ContainerDied","Data":"f59c2dd56e77af3941b696590dd366bc3337c69a5a4acd1d97d332e1fd0df64a"} Oct 30 01:15:19.761145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.761123 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" event={"ID":"33374e9d-1513-4645-a044-d979e11ac979","Type":"ContainerDied","Data":"05d7747d42b9afad847f149f58b99170f42ecbf4043a6cafc3820e4b15f5043a"} Oct 30 01:15:19.761259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.761212 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl" Oct 30 01:15:19.777835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.777757 4905 scope.go:122] "RemoveContainer" containerID="078c2f9204061006b0b05e5540de65953279a23499a779568e3ce9f78e93cde1" Oct 30 01:15:19.781958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.781932 4905 generic.go:356] "Generic (PLEG): container finished" podID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerID="d073d4511db0eac0be783ca9cedf80f27f48342e67256ec390f02998d32a7837" exitCode=60 Oct 30 01:15:19.782067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.781991 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerDied","Data":"d073d4511db0eac0be783ca9cedf80f27f48342e67256ec390f02998d32a7837"} Oct 30 01:15:19.782694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.782664 4905 scope.go:122] "RemoveContainer" containerID="d073d4511db0eac0be783ca9cedf80f27f48342e67256ec390f02998d32a7837" Oct 30 01:15:19.783291 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:19.783256 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8_odh-ai-gateway-infra(f1ff5453-b37c-496b-8abc-e95ff86338df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" Oct 30 01:15:19.790954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.790937 4905 scope.go:122] "RemoveContainer" containerID="6ce8d5bbf1af9958a426128e8c2508bc035584251a5b5e51015754bfd66b8e2d" Oct 30 01:15:19.796656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.796637 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg"] Oct 30 01:15:19.802033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.802012 4905 scope.go:122] "RemoveContainer" containerID="441ae06f9c4a5f65eb22878516adf101a81aedb8da795f7c287bb104ab2fb880" Oct 30 01:15:19.821297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.821268 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj"] Oct 30 01:15:19.828366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.826735 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-cfe8f0-29888715-9rsrj"] Oct 30 01:15:19.846805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.846780 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8"] Oct 30 01:15:19.885764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.885737 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k"] Oct 30 01:15:19.891205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.891179 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888715-6ds4k"] Oct 30 01:15:19.976563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.976480 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl"] Oct 30 01:15:19.980779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.980753 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95fe33-29888715-jj7fl"] Oct 30 01:15:19.987155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:19.987132 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:20.040967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.040897 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a4faf7-b736-44ef-bc82-195af1bf0af8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a4faf7-b736-44ef-bc82-195af1bf0af8-maas-api-service-ca\") pod \"54a4faf7-b736-44ef-bc82-195af1bf0af8\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " Oct 30 01:15:20.041232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.041212 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a4faf7-b736-44ef-bc82-195af1bf0af8-kube-api-access-g8qff\" (UniqueName: \"kubernetes.io/projected/54a4faf7-b736-44ef-bc82-195af1bf0af8-kube-api-access-g8qff\") pod \"54a4faf7-b736-44ef-bc82-195af1bf0af8\" (UID: \"54a4faf7-b736-44ef-bc82-195af1bf0af8\") " Oct 30 01:15:20.041314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.041271 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a4faf7-b736-44ef-bc82-195af1bf0af8-maas-api-service-ca" pod "54a4faf7-b736-44ef-bc82-195af1bf0af8" (UID: "54a4faf7-b736-44ef-bc82-195af1bf0af8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:20.041862 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.041841 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a4faf7-b736-44ef-bc82-195af1bf0af8-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.043566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.043500 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a4faf7-b736-44ef-bc82-195af1bf0af8-kube-api-access-g8qff" pod "54a4faf7-b736-44ef-bc82-195af1bf0af8" (UID: "54a4faf7-b736-44ef-bc82-195af1bf0af8"). InnerVolumeSpecName "kube-api-access-g8qff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:20.165640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.160070 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qff\" (UniqueName: \"kubernetes.io/projected/54a4faf7-b736-44ef-bc82-195af1bf0af8-kube-api-access-g8qff\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.802743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.802717 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" Oct 30 01:15:20.803157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.802750 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg" event={"ID":"54a4faf7-b736-44ef-bc82-195af1bf0af8","Type":"ContainerDied","Data":"f078323e1924f363f870b7b663651ea9882b9bf9ec192cd9bdc871e9d4a47eaa"} Oct 30 01:15:20.803157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.802814 4905 scope.go:122] "RemoveContainer" containerID="cafc098171720047ade7df6efd6b49b1e9cfa71c24668035fc05c8c8c7bd02a4" Oct 30 01:15:20.805273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.805244 4905 generic.go:356] "Generic (PLEG): container finished" podID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerID="711a68a6202f33230a872f118db46231d83bbbef0cedc9e9f7e26a1f269da561" exitCode=60 Oct 30 01:15:20.805391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.805317 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" event={"ID":"9a049cef-bb00-46d6-85e9-7fbd280b3d0e","Type":"ContainerDied","Data":"711a68a6202f33230a872f118db46231d83bbbef0cedc9e9f7e26a1f269da561"} Oct 30 01:15:20.805824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.805801 4905 scope.go:122] "RemoveContainer" containerID="711a68a6202f33230a872f118db46231d83bbbef0cedc9e9f7e26a1f269da561" Oct 30 01:15:20.806167 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:20.806143 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-lr27c_odh-ai-gateway-infra(9a049cef-bb00-46d6-85e9-7fbd280b3d0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" Oct 30 01:15:20.816735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.816706 4905 scope.go:122] "RemoveContainer" containerID="55fbd97c063191c875d8e1e0dcba5aeb2c554c3896f477218bc380415b038457" Oct 30 01:15:20.887940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.887909 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg"] Oct 30 01:15:20.892161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:20.892134 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b973b-29888715-zsftg"] Oct 30 01:15:21.033663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.033637 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:21.175376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.175292 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ff5453-b37c-496b-8abc-e95ff86338df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff5453-b37c-496b-8abc-e95ff86338df-maas-api-service-ca\") pod \"f1ff5453-b37c-496b-8abc-e95ff86338df\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " Oct 30 01:15:21.175539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.175375 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ff5453-b37c-496b-8abc-e95ff86338df-kube-api-access-4blmn\" (UniqueName: \"kubernetes.io/projected/f1ff5453-b37c-496b-8abc-e95ff86338df-kube-api-access-4blmn\") pod \"f1ff5453-b37c-496b-8abc-e95ff86338df\" (UID: \"f1ff5453-b37c-496b-8abc-e95ff86338df\") " Oct 30 01:15:21.175673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.175653 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ff5453-b37c-496b-8abc-e95ff86338df-maas-api-service-ca" pod "f1ff5453-b37c-496b-8abc-e95ff86338df" (UID: "f1ff5453-b37c-496b-8abc-e95ff86338df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:21.176039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.176015 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff5453-b37c-496b-8abc-e95ff86338df-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.177683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.177661 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ff5453-b37c-496b-8abc-e95ff86338df-kube-api-access-4blmn" pod "f1ff5453-b37c-496b-8abc-e95ff86338df" (UID: "f1ff5453-b37c-496b-8abc-e95ff86338df"). InnerVolumeSpecName "kube-api-access-4blmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:21.193644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.193620 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" path="/var/lib/kubelet/pods/31e0c08a-049e-484e-aab6-ddeeed75c2f1/volumes" Oct 30 01:15:21.194087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.194072 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33374e9d-1513-4645-a044-d979e11ac979" path="/var/lib/kubelet/pods/33374e9d-1513-4645-a044-d979e11ac979/volumes" Oct 30 01:15:21.194476 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.194463 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" path="/var/lib/kubelet/pods/54a4faf7-b736-44ef-bc82-195af1bf0af8/volumes" Oct 30 01:15:21.195212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.195196 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" path="/var/lib/kubelet/pods/f379fc6d-07de-491f-8c4b-210a84498a91/volumes" Oct 30 01:15:21.277192 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.277166 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4blmn\" (UniqueName: \"kubernetes.io/projected/f1ff5453-b37c-496b-8abc-e95ff86338df-kube-api-access-4blmn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.899173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.899140 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" Oct 30 01:15:21.933205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.900115 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8" event={"ID":"f1ff5453-b37c-496b-8abc-e95ff86338df","Type":"ContainerDied","Data":"51aa4e06c3fc903ea6637868034eb219efc487d84fe4fb10aa993e969d4e9db0"} Oct 30 01:15:21.933205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.932844 4905 scope.go:122] "RemoveContainer" containerID="d073d4511db0eac0be783ca9cedf80f27f48342e67256ec390f02998d32a7837" Oct 30 01:15:21.947234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:21.947204 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c"] Oct 30 01:15:22.050304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.045677 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8"] Oct 30 01:15:22.053691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.051462 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24d5ce-29888715-dgnt8"] Oct 30 01:15:22.137090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.137065 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:22.206733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.206648 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-kube-api-access-t79ws\" (UniqueName: \"kubernetes.io/projected/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-kube-api-access-t79ws\") pod \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " Oct 30 01:15:22.206733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.206720 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-maas-api-service-ca\") pod \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\" (UID: \"9a049cef-bb00-46d6-85e9-7fbd280b3d0e\") " Oct 30 01:15:22.207134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.207102 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-maas-api-service-ca" pod "9a049cef-bb00-46d6-85e9-7fbd280b3d0e" (UID: "9a049cef-bb00-46d6-85e9-7fbd280b3d0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.207804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.207769 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.209194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.209168 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-kube-api-access-t79ws" pod "9a049cef-bb00-46d6-85e9-7fbd280b3d0e" (UID: "9a049cef-bb00-46d6-85e9-7fbd280b3d0e"). InnerVolumeSpecName "kube-api-access-t79ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.309783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.309757 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79ws\" (UniqueName: \"kubernetes.io/projected/9a049cef-bb00-46d6-85e9-7fbd280b3d0e-kube-api-access-t79ws\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.949646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.949617 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" Oct 30 01:15:22.950131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.949662 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c" event={"ID":"9a049cef-bb00-46d6-85e9-7fbd280b3d0e","Type":"ContainerDied","Data":"70344984be68d10efe887a489c63097c2c5d52436aa244b2b7251c0249fdbf75"} Oct 30 01:15:22.950131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:22.949731 4905 scope.go:122] "RemoveContainer" containerID="711a68a6202f33230a872f118db46231d83bbbef0cedc9e9f7e26a1f269da561" Oct 30 01:15:23.006972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:23.006944 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c"] Oct 30 01:15:23.011229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:23.011206 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-lr27c"] Oct 30 01:15:23.193018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:23.192986 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" path="/var/lib/kubelet/pods/9a049cef-bb00-46d6-85e9-7fbd280b3d0e/volumes" Oct 30 01:15:23.193427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:23.193414 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" path="/var/lib/kubelet/pods/f1ff5453-b37c-496b-8abc-e95ff86338df/volumes" Oct 30 01:15:38.120497 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:38.120411 4905 generic.go:356] "Generic (PLEG): container finished" podID="17922593-835e-4bd3-9e76-7510d501815f" containerID="d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c" exitCode=0 Oct 30 01:15:38.121051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:38.120491 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" event={"ID":"17922593-835e-4bd3-9e76-7510d501815f","Type":"ContainerDied","Data":"d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c"} Oct 30 01:15:38.121051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:38.121027 4905 scope.go:122] "RemoveContainer" containerID="d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c" Oct 30 01:15:38.122772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:38.122740 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:15:38.122897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:38.122823 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:15:38.122897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:38.122885 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:15:38.498422 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:38.498380 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:15:39.156051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:39.156006 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" event={"ID":"17922593-835e-4bd3-9e76-7510d501815f","Type":"ContainerStarted","Data":"00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e"} Oct 30 01:15:39.405074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:39.405044 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:15:39.406577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:39.406497 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 01:15:39.406724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:39.406584 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 01:15:40.406038 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:40.406010 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 01:15:40.406460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:40.406056 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 01:15:41.405343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:41.405310 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 01:15:41.405585 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:41.405371 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 01:15:42.406950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:42.406901 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 01:15:42.407445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:42.406976 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 01:15:42.465171 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:15:42.465135 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:15:43.405632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:43.405605 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 01:15:43.405871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:43.405848 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 01:15:44.410812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:44.410781 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:15:44.411248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:44.411230 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:15:44.411932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:15:44.411908 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:16:09.433083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.433053 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm"] Oct 30 01:16:09.434255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434230 4905 cpu_manager.go:414] "containerMap: removing container" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:09.434255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434251 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434271 4905 cpu_manager.go:414] "containerMap: removing container" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434276 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434284 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434289 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434298 4905 cpu_manager.go:414] "containerMap: removing container" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434303 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434310 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434314 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434325 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434329 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434337 4905 cpu_manager.go:414] "containerMap: removing container" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434350 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434363 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434367 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434379 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434383 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434392 4905 cpu_manager.go:414] "containerMap: removing container" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434396 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434403 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434407 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434432 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434439 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434459 4905 cpu_manager.go:414] "containerMap: removing container" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434464 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434474 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434478 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434490 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:09.434478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434494 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434508 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434535 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434542 4905 cpu_manager.go:414] "containerMap: removing container" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434547 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434556 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434559 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434568 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434572 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434691 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434698 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434705 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434711 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434717 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434722 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434726 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434731 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434738 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434748 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434756 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434762 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434767 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434773 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434779 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434784 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434791 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434796 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434802 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434808 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434814 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434825 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434833 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:09.436032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.434840 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:09.438908 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.438887 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.459962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.459918 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-ad00a0-openshift-default-dockercfg-w2ff8\"" Oct 30 01:16:09.474595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.474512 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm"] Oct 30 01:16:09.568300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568266 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568317 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-certs\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568393 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-credential-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568464 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-data\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568485 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istiod-ca-cert\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568561 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-envoy\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568583 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-token\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568626 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-podinfo\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.568720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.568713 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh24z\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-kube-api-access-wh24z\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.677945 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678022 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-certs\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678050 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-credential-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678115 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-data\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678145 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istiod-ca-cert\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678212 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-envoy\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678249 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-token\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678291 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-podinfo\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.678543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.678362 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh24z\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-kube-api-access-wh24z\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.690593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.682941 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-data\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.690593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.683190 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.690593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.683409 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-certs\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.690593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.683684 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-credential-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.727544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.720853 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istiod-ca-cert\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.727544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.721044 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-envoy\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.742542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.733506 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-podinfo\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.747546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.747317 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-token\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:09.795590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:09.789218 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh24z\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-kube-api-access-wh24z\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:10.057082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.057050 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:10.383845 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:16:10.383815 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e381e2_5b6b_49af_8fe2_5ba37a189955.slice/crio-9d99a8de0fe67fa398c23530469b52b5ec35556e040678000c36e8bcfc449c32 WatchSource:0}: Error finding container 9d99a8de0fe67fa398c23530469b52b5ec35556e040678000c36e8bcfc449c32: Status 404 returned error can't find the container with id 9d99a8de0fe67fa398c23530469b52b5ec35556e040678000c36e8bcfc449c32 Oct 30 01:16:10.386140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.386117 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm"] Oct 30 01:16:10.387188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.387156 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:16:10.387266 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.387219 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:16:10.387266 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.387251 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:16:10.737571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.736968 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" event={"ID":"e5e381e2-5b6b-49af-8fe2-5ba37a189955","Type":"ContainerStarted","Data":"8251a58ab9944e283df7ed3eb80d2d678d314344ce1baa56d14ce1953d3de845"} Oct 30 01:16:10.737571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.737041 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" event={"ID":"e5e381e2-5b6b-49af-8fe2-5ba37a189955","Type":"ContainerStarted","Data":"9d99a8de0fe67fa398c23530469b52b5ec35556e040678000c36e8bcfc449c32"} Oct 30 01:16:10.785937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:10.785884 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" podStartSLOduration=1.785870567 podStartE2EDuration="1.785870567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:10.780821847 +0000 UTC m=+2255.723995039" watchObservedRunningTime="2026-10-30 01:16:10.785870567 +0000 UTC m=+2255.729043758" Oct 30 01:16:11.057398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:11.057314 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:11.063769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:11.063746 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:11.759206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:11.759175 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:11.760387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:11.760361 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:16:12.416022 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:16:12.415982 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:16:15.329676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.329642 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9"] Oct 30 01:16:15.330434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330415 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:15.330483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330436 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40fd573-8d64-425e-9134-054f7440e187" containerName="cleanup" Oct 30 01:16:15.330483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330455 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:15.330483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330460 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4b96d8-ffc3-4e55-83de-4a3b8d216b56" containerName="cleanup" Oct 30 01:16:15.330483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330470 4905 cpu_manager.go:414] "containerMap: removing container" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:15.330483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330475 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a4faf7-b736-44ef-bc82-195af1bf0af8" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330487 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330492 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a049cef-bb00-46d6-85e9-7fbd280b3d0e" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330509 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330528 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330539 4905 cpu_manager.go:414] "containerMap: removing container" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330545 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330557 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330562 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f379fc6d-07de-491f-8c4b-210a84498a91" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330570 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:15.330660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330574 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b168dc55-dfba-45d0-b281-80390072df92" containerName="cleanup" Oct 30 01:16:15.330975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330698 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e0c08a-049e-484e-aab6-ddeeed75c2f1" containerName="cleanup" Oct 30 01:16:15.330975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330709 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="33374e9d-1513-4645-a044-d979e11ac979" containerName="cleanup" Oct 30 01:16:15.330975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.330714 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ff5453-b37c-496b-8abc-e95ff86338df" containerName="cleanup" Oct 30 01:16:15.334761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.334735 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.355034 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.355013 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9"] Oct 30 01:16:15.369800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.369713 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-ad00a0\"" Oct 30 01:16:15.391951 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.391919 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl"] Oct 30 01:16:15.397821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.397798 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:15.415107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.415079 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl"] Oct 30 01:16:15.415372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.415353 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-ad00a0-dockercfg-bnqgr\"" Oct 30 01:16:15.500584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.498595 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e8119b-10cc-4552-ae70-5b621711f586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:15.500584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.498725 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b802cb-c008-4902-b352-fc95ca96d5a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.500584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.498836 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64wz\" (UniqueName: \"kubernetes.io/projected/09b802cb-c008-4902-b352-fc95ca96d5a5-kube-api-access-f64wz\") pod \"payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.500584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.498899 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wfb\" (UniqueName: \"kubernetes.io/projected/46e8119b-10cc-4552-ae70-5b621711f586-kube-api-access-w6wfb\") pod \"payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:15.602649 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.602548 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64wz\" (UniqueName: \"kubernetes.io/projected/09b802cb-c008-4902-b352-fc95ca96d5a5-kube-api-access-f64wz\") pod \"payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.602931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.602910 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wfb\" (UniqueName: \"kubernetes.io/projected/46e8119b-10cc-4552-ae70-5b621711f586-kube-api-access-w6wfb\") pod \"payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:15.603239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.603222 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e8119b-10cc-4552-ae70-5b621711f586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:15.603441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.603426 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b802cb-c008-4902-b352-fc95ca96d5a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.608026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.607996 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e8119b-10cc-4552-ae70-5b621711f586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:15.609638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.609583 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b802cb-c008-4902-b352-fc95ca96d5a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.650635 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.650599 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64wz\" (UniqueName: \"kubernetes.io/projected/09b802cb-c008-4902-b352-fc95ca96d5a5-kube-api-access-f64wz\") pod \"payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.651107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.651082 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:15.653025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.653000 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wfb\" (UniqueName: \"kubernetes.io/projected/46e8119b-10cc-4552-ae70-5b621711f586-kube-api-access-w6wfb\") pod \"payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:15.721569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:15.720156 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:16.008022 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:16:16.007994 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b802cb_c008_4902_b352_fc95ca96d5a5.slice/crio-17207606109c0e810ef939e3acec1051a3b2b9772d3558b6a80707eb1f317368 WatchSource:0}: Error finding container 17207606109c0e810ef939e3acec1051a3b2b9772d3558b6a80707eb1f317368: Status 404 returned error can't find the container with id 17207606109c0e810ef939e3acec1051a3b2b9772d3558b6a80707eb1f317368 Oct 30 01:16:16.010784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.010757 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9"] Oct 30 01:16:16.073325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.073293 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl"] Oct 30 01:16:16.828304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.828267 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" event={"ID":"46e8119b-10cc-4552-ae70-5b621711f586","Type":"ContainerStarted","Data":"018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225"} Oct 30 01:16:16.828876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.828347 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" event={"ID":"46e8119b-10cc-4552-ae70-5b621711f586","Type":"ContainerStarted","Data":"037ca05607df9ca843398fd1679856f243685881f45a28ba3ea4e98f8eced614"} Oct 30 01:16:16.828876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.828390 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:16.830324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.830292 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" event={"ID":"09b802cb-c008-4902-b352-fc95ca96d5a5","Type":"ContainerStarted","Data":"9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f"} Oct 30 01:16:16.830489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.830358 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:16.830489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.830377 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" event={"ID":"09b802cb-c008-4902-b352-fc95ca96d5a5","Type":"ContainerStarted","Data":"17207606109c0e810ef939e3acec1051a3b2b9772d3558b6a80707eb1f317368"} Oct 30 01:16:16.874471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.874431 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" podStartSLOduration=1.874418844 podStartE2EDuration="1.874418844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:16.871746266 +0000 UTC m=+2261.814919458" watchObservedRunningTime="2026-10-30 01:16:16.874418844 +0000 UTC m=+2261.817592035" Oct 30 01:16:16.912535 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:16.911379 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" podStartSLOduration=1.911363799 podStartE2EDuration="1.911363799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:16.910940597 +0000 UTC m=+2261.854113790" watchObservedRunningTime="2026-10-30 01:16:16.911363799 +0000 UTC m=+2261.854536992" Oct 30 01:16:27.841493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:27.841448 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:16:27.842079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:27.841582 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:16:36.348492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:36.348460 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr"] Oct 30 01:16:42.177864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:42.177816 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" podUID="fd2d942e-d568-4861-bf79-bd8799145eda" containerName="payload-processing" containerID="cri-o://4341babe456c8ab8a344c715035ed66a45665d5c3691fdabc13f8d3bb06407d3" gracePeriod=25 Oct 30 01:16:42.417406 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:16:42.417370 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:16:48.519920 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.519833 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" podUID="fd2d942e-d568-4861-bf79-bd8799145eda" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:48.724594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.724561 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv"] Oct 30 01:16:48.730365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.730344 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.749315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.749290 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv"] Oct 30 01:16:48.757942 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.757912 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-ad00a0-serving-cert\"" Oct 30 01:16:48.844186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.844104 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.844186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.844166 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada8efa9-0769-4886-9d93-14b541f44f69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.844390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.844313 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.844470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.844446 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkpt\" (UniqueName: \"kubernetes.io/projected/ada8efa9-0769-4886-9d93-14b541f44f69-kube-api-access-6rkpt\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.946647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.946603 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkpt\" (UniqueName: \"kubernetes.io/projected/ada8efa9-0769-4886-9d93-14b541f44f69-kube-api-access-6rkpt\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.947197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.947167 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.968543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.947396 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada8efa9-0769-4886-9d93-14b541f44f69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.969013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.968984 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.969902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.969865 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.971577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.951243 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada8efa9-0769-4886-9d93-14b541f44f69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.982823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.982791 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:48.997471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:48.997271 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkpt\" (UniqueName: \"kubernetes.io/projected/ada8efa9-0769-4886-9d93-14b541f44f69-kube-api-access-6rkpt\") pod \"maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:49.045107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:49.045069 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:49.365078 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:16:49.365050 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada8efa9_0769_4886_9d93_14b541f44f69.slice/crio-a8a245f9e7620f7aee4e2e357421432a15c52655b438c30a498a474b517c8de3 WatchSource:0}: Error finding container a8a245f9e7620f7aee4e2e357421432a15c52655b438c30a498a474b517c8de3: Status 404 returned error can't find the container with id a8a245f9e7620f7aee4e2e357421432a15c52655b438c30a498a474b517c8de3 Oct 30 01:16:49.366193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:49.366172 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv"] Oct 30 01:16:49.366459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:49.366444 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:16:50.138992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.138909 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6"] Oct 30 01:16:50.139431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.139171 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" podUID="9f65e2dc-800e-4f18-8961-2b05692cd66a" containerName="maas-api" containerID="cri-o://9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b" gracePeriod=30 Oct 30 01:16:50.333831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.333797 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" event={"ID":"ada8efa9-0769-4886-9d93-14b541f44f69","Type":"ContainerStarted","Data":"7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060"} Oct 30 01:16:50.334011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.333868 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:50.334011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.333887 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" event={"ID":"ada8efa9-0769-4886-9d93-14b541f44f69","Type":"ContainerStarted","Data":"a8a245f9e7620f7aee4e2e357421432a15c52655b438c30a498a474b517c8de3"} Oct 30 01:16:50.429787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.421626 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" podStartSLOduration=1.993648978 podStartE2EDuration="2.421605697s" totalImagesPullingTime="427.956719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:50.37796789 +0000 UTC m=+2295.321141081" watchObservedRunningTime="2026-10-30 01:16:50.421605697 +0000 UTC m=+2295.364778889" Oct 30 01:16:50.872967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.872931 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:16:50.920183 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.920148 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f65e2dc-800e-4f18-8961-2b05692cd66a-kube-api-access-xnrnh\" (UniqueName: \"kubernetes.io/projected/9f65e2dc-800e-4f18-8961-2b05692cd66a-kube-api-access-xnrnh\") pod \"9f65e2dc-800e-4f18-8961-2b05692cd66a\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " Oct 30 01:16:50.920405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.920390 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f65e2dc-800e-4f18-8961-2b05692cd66a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f65e2dc-800e-4f18-8961-2b05692cd66a-openshift-service-ca\") pod \"9f65e2dc-800e-4f18-8961-2b05692cd66a\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " Oct 30 01:16:50.920491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.920441 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls\") pod \"9f65e2dc-800e-4f18-8961-2b05692cd66a\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " Oct 30 01:16:50.920568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.920502 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-metrics-certs\") pod \"9f65e2dc-800e-4f18-8961-2b05692cd66a\" (UID: \"9f65e2dc-800e-4f18-8961-2b05692cd66a\") " Oct 30 01:16:50.920847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.920775 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f65e2dc-800e-4f18-8961-2b05692cd66a-openshift-service-ca" pod "9f65e2dc-800e-4f18-8961-2b05692cd66a" (UID: "9f65e2dc-800e-4f18-8961-2b05692cd66a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:50.921254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.921219 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f65e2dc-800e-4f18-8961-2b05692cd66a-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:16:50.922783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.922757 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f65e2dc-800e-4f18-8961-2b05692cd66a-kube-api-access-xnrnh" pod "9f65e2dc-800e-4f18-8961-2b05692cd66a" (UID: "9f65e2dc-800e-4f18-8961-2b05692cd66a"). InnerVolumeSpecName "kube-api-access-xnrnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:50.922900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.922799 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls" pod "9f65e2dc-800e-4f18-8961-2b05692cd66a" (UID: "9f65e2dc-800e-4f18-8961-2b05692cd66a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:50.923107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:50.923086 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-metrics-certs" pod "9f65e2dc-800e-4f18-8961-2b05692cd66a" (UID: "9f65e2dc-800e-4f18-8961-2b05692cd66a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:51.022354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.022287 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:16:51.022354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.022311 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f65e2dc-800e-4f18-8961-2b05692cd66a-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:16:51.022354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.022322 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnrnh\" (UniqueName: \"kubernetes.io/projected/9f65e2dc-800e-4f18-8961-2b05692cd66a-kube-api-access-xnrnh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:16:51.344141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.344057 4905 generic.go:356] "Generic (PLEG): container finished" podID="9f65e2dc-800e-4f18-8961-2b05692cd66a" containerID="9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b" exitCode=0 Oct 30 01:16:51.344580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.344161 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" Oct 30 01:16:51.344580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.344165 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" event={"ID":"9f65e2dc-800e-4f18-8961-2b05692cd66a","Type":"ContainerDied","Data":"9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b"} Oct 30 01:16:51.344580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.344229 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6" event={"ID":"9f65e2dc-800e-4f18-8961-2b05692cd66a","Type":"ContainerDied","Data":"6340b9a069a3be049e13ab6d2f9392cfd8d0e857897bacf07191f29651ce7f6e"} Oct 30 01:16:51.344580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.344259 4905 scope.go:122] "RemoveContainer" containerID="9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b" Oct 30 01:16:51.356253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.356233 4905 scope.go:122] "RemoveContainer" containerID="9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b" Oct 30 01:16:51.356564 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:16:51.356541 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b\": container with ID starting with 9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b not found: ID does not exist" containerID="9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b" Oct 30 01:16:51.356666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.356567 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b"} err="failed to get container status \"9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b\": rpc error: code = NotFound desc = could not find container \"9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b\": container with ID starting with 9108c25162b96099590ab892f60602dec5621c882dce7d768e411f1e6ae1e57b not found: ID does not exist" Oct 30 01:16:51.403078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.403051 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6"] Oct 30 01:16:51.406954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:51.406928 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b312f7-7b59b68fb6-ncct6"] Oct 30 01:16:53.192943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:53.192905 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f65e2dc-800e-4f18-8961-2b05692cd66a" path="/var/lib/kubelet/pods/9f65e2dc-800e-4f18-8961-2b05692cd66a/volumes" Oct 30 01:16:56.351122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:56.351095 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:16:57.595477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:57.595445 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr"] Oct 30 01:16:57.595914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:57.595712 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" containerID="cri-o://58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09" gracePeriod=30 Oct 30 01:16:58.520250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:58.520218 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" podUID="fd2d942e-d568-4861-bf79-bd8799145eda" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:59.235563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:16:59.224895 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74"] Oct 30 01:17:02.512290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.512258 4905 generic.go:356] "Generic (PLEG): container finished" podID="fd2d942e-d568-4861-bf79-bd8799145eda" containerID="4341babe456c8ab8a344c715035ed66a45665d5c3691fdabc13f8d3bb06407d3" exitCode=0 Oct 30 01:17:02.512652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.512329 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" event={"ID":"fd2d942e-d568-4861-bf79-bd8799145eda","Type":"ContainerDied","Data":"4341babe456c8ab8a344c715035ed66a45665d5c3691fdabc13f8d3bb06407d3"} Oct 30 01:17:02.512652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.512380 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" event={"ID":"fd2d942e-d568-4861-bf79-bd8799145eda","Type":"ContainerDied","Data":"3f6312adb6ece0aac482602322c225c35afdab6e4132093c85ada20f4b42f6af"} Oct 30 01:17:02.512652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.512395 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6312adb6ece0aac482602322c225c35afdab6e4132093c85ada20f4b42f6af" Oct 30 01:17:02.526818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.526794 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:17:02.639008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.638979 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2d942e-d568-4861-bf79-bd8799145eda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2d942e-d568-4861-bf79-bd8799145eda-plugins-config-volume\") pod \"fd2d942e-d568-4861-bf79-bd8799145eda\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " Oct 30 01:17:02.639145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.639041 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2d942e-d568-4861-bf79-bd8799145eda-kube-api-access-gzkcl\" (UniqueName: \"kubernetes.io/projected/fd2d942e-d568-4861-bf79-bd8799145eda-kube-api-access-gzkcl\") pod \"fd2d942e-d568-4861-bf79-bd8799145eda\" (UID: \"fd2d942e-d568-4861-bf79-bd8799145eda\") " Oct 30 01:17:02.639436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.639410 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2d942e-d568-4861-bf79-bd8799145eda-plugins-config-volume" pod "fd2d942e-d568-4861-bf79-bd8799145eda" (UID: "fd2d942e-d568-4861-bf79-bd8799145eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:02.639688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.639668 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2d942e-d568-4861-bf79-bd8799145eda-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:02.641845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.641820 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2d942e-d568-4861-bf79-bd8799145eda-kube-api-access-gzkcl" pod "fd2d942e-d568-4861-bf79-bd8799145eda" (UID: "fd2d942e-d568-4861-bf79-bd8799145eda"). InnerVolumeSpecName "kube-api-access-gzkcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:02.741464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.741366 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzkcl\" (UniqueName: \"kubernetes.io/projected/fd2d942e-d568-4861-bf79-bd8799145eda-kube-api-access-gzkcl\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:02.885975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.885952 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:17:02.944965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.944933 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istiod-ca-cert\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.944992 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-socket\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945066 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-data\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945284 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945267 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-certs\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945311 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-envoy\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945337 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istiod-ca-cert" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:02.945399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945368 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-podinfo\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945369 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-socket" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:02.945604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945411 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-kube-api-access-sc98r\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-kube-api-access-sc98r\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945532 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-token\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945563 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-credential-socket\") pod \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\" (UID: \"2ff5b4b8-e670-4715-b021-f2334afdbe4f\") " Oct 30 01:17:02.945768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.945610 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-data" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:02.946424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.946094 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-credential-socket" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:02.946424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.946214 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-certs" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:02.946424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.946346 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:02.946424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.946364 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:02.946424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.946376 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:02.946424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.946389 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:02.946424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.946402 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:02.948554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.948502 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-kube-api-access-sc98r" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "kube-api-access-sc98r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:02.948749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.948725 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-envoy" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:02.949803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.949781 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-token" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:02.950836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:02.950795 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-podinfo" pod "2ff5b4b8-e670-4715-b021-f2334afdbe4f" (UID: "2ff5b4b8-e670-4715-b021-f2334afdbe4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:03.048314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.048236 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:03.048314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.048264 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:03.048314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.048274 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc98r\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-kube-api-access-sc98r\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:03.048314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.048284 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5b4b8-e670-4715-b021-f2334afdbe4f-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:03.521122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.521090 4905 generic.go:356] "Generic (PLEG): container finished" podID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerID="58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09" exitCode=0 Oct 30 01:17:03.521648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.521136 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" Oct 30 01:17:03.521648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.521177 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" event={"ID":"2ff5b4b8-e670-4715-b021-f2334afdbe4f","Type":"ContainerDied","Data":"58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09"} Oct 30 01:17:03.521648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.521243 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr" event={"ID":"2ff5b4b8-e670-4715-b021-f2334afdbe4f","Type":"ContainerDied","Data":"e64168473e46c1ed13c48002b337cf17f74f002214b5889cd5c5bd82ee728e3b"} Oct 30 01:17:03.521648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.521273 4905 scope.go:122] "RemoveContainer" containerID="58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09" Oct 30 01:17:03.521648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.521283 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr" Oct 30 01:17:03.533233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.533206 4905 scope.go:122] "RemoveContainer" containerID="58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09" Oct 30 01:17:03.533571 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:17:03.533545 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09\": container with ID starting with 58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09 not found: ID does not exist" containerID="58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09" Oct 30 01:17:03.533643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.533582 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09"} err="failed to get container status \"58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09\": rpc error: code = NotFound desc = could not find container \"58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09\": container with ID starting with 58d5de0d65011ebca2403b853a70719427df3439077c042dbae5646fd9aadc09 not found: ID does not exist" Oct 30 01:17:03.600874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.600845 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr"] Oct 30 01:17:03.607011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.606983 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b312f7-8574f8dfdb-mzjsr"] Oct 30 01:17:03.643696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.643657 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr"] Oct 30 01:17:03.647444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:03.647416 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-hbcxr"] Oct 30 01:17:05.191795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:05.191762 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" path="/var/lib/kubelet/pods/2ff5b4b8-e670-4715-b021-f2334afdbe4f/volumes" Oct 30 01:17:05.192177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:05.192150 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2d942e-d568-4861-bf79-bd8799145eda" path="/var/lib/kubelet/pods/fd2d942e-d568-4861-bf79-bd8799145eda/volumes" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.386554 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5"] Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.387714 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9f65e2dc-800e-4f18-8961-2b05692cd66a" containerName="maas-api" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.387734 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f65e2dc-800e-4f18-8961-2b05692cd66a" containerName="maas-api" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.387786 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fd2d942e-d568-4861-bf79-bd8799145eda" containerName="payload-processing" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.387796 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2d942e-d568-4861-bf79-bd8799145eda" containerName="payload-processing" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.387855 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.387862 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.388016 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2d942e-d568-4861-bf79-bd8799145eda" containerName="payload-processing" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.388040 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff5b4b8-e670-4715-b021-f2334afdbe4f" containerName="istio-proxy" Oct 30 01:17:07.390543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.388054 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f65e2dc-800e-4f18-8961-2b05692cd66a" containerName="maas-api" Oct 30 01:17:07.394902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.394877 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.414573 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.413186 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5"] Oct 30 01:17:07.420574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.420495 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f461c3-openshift-default-dockercfg-274nh\"" Oct 30 01:17:07.541608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.526424 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istiod-ca-cert\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.542270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.542236 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-podinfo\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.542664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.542641 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-socket\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.542844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.542827 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-certs\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.543138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.543118 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-data\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.563544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.543345 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-envoy\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.564001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.563968 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-credential-socket\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.564262 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.564242 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blbfc\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-kube-api-access-blbfc\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.564553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.564509 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-token\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.668086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.667985 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-data\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.668350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.668331 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-envoy\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.668545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.668506 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-credential-socket\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.684545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.669722 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbfc\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-kube-api-access-blbfc\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.716575 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-token\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.716738 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istiod-ca-cert\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.716814 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-envoy\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.716858 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-podinfo\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.716938 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-socket\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.716963 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-certs\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.717872 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-socket\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.718228 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istiod-ca-cert\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.669654 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-data\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.669056 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-credential-socket\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.720610 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-certs\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.725546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.723745 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-podinfo\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.728263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.727342 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbfc\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-kube-api-access-blbfc\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:07.775536 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:07.775485 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-token\") pod \"e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:08.016532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.016489 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:08.362478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.362438 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:08.362613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.362510 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:08.362613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.362560 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:08.366442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.366422 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5"] Oct 30 01:17:08.607453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.607357 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj"] Oct 30 01:17:08.668634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.665635 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" event={"ID":"52df6f9a-b866-4fa6-8b50-16606cb8e1ac","Type":"ContainerStarted","Data":"a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a"} Oct 30 01:17:08.668634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.665699 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" event={"ID":"52df6f9a-b866-4fa6-8b50-16606cb8e1ac","Type":"ContainerStarted","Data":"aba875438e63146037091f78e815b94996961f110a2db1f56c72b250200d46c4"} Oct 30 01:17:08.670621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.670591 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv"] Oct 30 01:17:08.776545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:08.775875 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podStartSLOduration=1.775856576 podStartE2EDuration="1.775856576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:08.764385392 +0000 UTC m=+2313.707558584" watchObservedRunningTime="2026-10-30 01:17:08.775856576 +0000 UTC m=+2313.719029770" Oct 30 01:17:09.019583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:09.019140 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:09.019583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:09.019283 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:17:09.019583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:09.019326 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 01:17:10.062548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:10.056045 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:10.719267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:10.719198 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:10.721949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:10.721778 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:17:12.416703 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:17:12.416648 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:17:13.579981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.579944 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt"] Oct 30 01:17:13.585412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.585387 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:13.602040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.602011 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt"] Oct 30 01:17:13.608449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.608388 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" podUID="f2b68575-298b-487d-9112-6f387aefaa50" containerName="payload-processing" containerID="cri-o://d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828" gracePeriod=25 Oct 30 01:17:13.612562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.612509 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f461c3\"" Oct 30 01:17:13.612841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.612819 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f461c3-dockercfg-wtznb\"" Oct 30 01:17:13.622210 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.622183 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx"] Oct 30 01:17:13.668223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.668193 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:13.673595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.672894 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" podUID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" containerName="payload-pre-processing" containerID="cri-o://90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43" gracePeriod=25 Oct 30 01:17:13.725954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.725913 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqd2j\" (UniqueName: \"kubernetes.io/projected/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-kube-api-access-xqd2j\") pod \"payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:13.743758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.726035 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:13.918659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.848838 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqd2j\" (UniqueName: \"kubernetes.io/projected/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-kube-api-access-xqd2j\") pod \"payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:13.918659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.848937 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:13.918659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.849083 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8674f5b-07a1-4d8b-abee-a37cca9206ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:13.918659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.849257 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859gl\" (UniqueName: \"kubernetes.io/projected/a8674f5b-07a1-4d8b-abee-a37cca9206ad-kube-api-access-859gl\") pod \"payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:13.918659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.850473 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:13.997173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.997132 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8674f5b-07a1-4d8b-abee-a37cca9206ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:13.997679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:13.997647 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859gl\" (UniqueName: \"kubernetes.io/projected/a8674f5b-07a1-4d8b-abee-a37cca9206ad-kube-api-access-859gl\") pod \"payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:14.043566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.030189 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8674f5b-07a1-4d8b-abee-a37cca9206ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:14.102941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.099604 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859gl\" (UniqueName: \"kubernetes.io/projected/a8674f5b-07a1-4d8b-abee-a37cca9206ad-kube-api-access-859gl\") pod \"payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:14.115144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.113293 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx"] Oct 30 01:17:14.119995 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.119969 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqd2j\" (UniqueName: \"kubernetes.io/projected/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-kube-api-access-xqd2j\") pod \"payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:14.184904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.181470 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf"] Oct 30 01:17:14.187771 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.187360 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.202175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.202129 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:14.203383 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.203223 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf"] Oct 30 01:17:14.203939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.203905 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1075cb-openshift-default-dockercfg-s2jxj\"" Oct 30 01:17:14.273605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.271074 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" podUID="8144991b-f4ea-4314-962e-376ff644a7d3" containerName="main" containerID="cri-o://1f15df15dcdde245046b19abceac281e0df119d150902ed05f00f31cfbb4139e" gracePeriod=45 Oct 30 01:17:14.316488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.316455 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-podinfo\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.316701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.316681 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5zll\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-kube-api-access-k5zll\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.316751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.316729 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-socket\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.316929 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.316915 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-credential-socket\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.317001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.316986 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-token\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.317045 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.317032 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-envoy\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.317107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.317093 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-data\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.317195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.317180 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-certs\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.317237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.317211 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a1e07-822c-443b-bf43-293a1455bc8b-istiod-ca-cert\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432200 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-podinfo\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432355 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zll\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-kube-api-access-k5zll\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432393 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-socket\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432537 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-credential-socket\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432609 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-token\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432643 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-envoy\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432696 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-data\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432775 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-certs\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.432801 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a1e07-822c-443b-bf43-293a1455bc8b-istiod-ca-cert\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.438594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.433709 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a1e07-822c-443b-bf43-293a1455bc8b-istiod-ca-cert\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.475562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.460909 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-socket\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.475562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.461129 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-credential-socket\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.475562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.461314 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-data\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.538372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.477345 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-certs\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.552037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.544208 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:14.613242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.606314 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zll\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-kube-api-access-k5zll\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.658320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.658258 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-token\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.707717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.703592 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-podinfo\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.707717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.706698 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" podUID="8144991b-f4ea-4314-962e-376ff644a7d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.250:8000/ready\": dial tcp 10.132.0.250:8000: connect: connection refused" Oct 30 01:17:14.707717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.707637 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-envoy\") pod \"e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.795954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.795919 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:17:14.814119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.812896 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt"] Oct 30 01:17:14.814119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.813157 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:14.815239 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:17:14.815211 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8674f5b_07a1_4d8b_abee_a37cca9206ad.slice/crio-a7ad9e475f9ad77bed160b7bd1150c17fefb8c93a48b14e051855d721ced898d WatchSource:0}: Error finding container a7ad9e475f9ad77bed160b7bd1150c17fefb8c93a48b14e051855d721ced898d: Status 404 returned error can't find the container with id a7ad9e475f9ad77bed160b7bd1150c17fefb8c93a48b14e051855d721ced898d Oct 30 01:17:14.894575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.892423 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b68575-298b-487d-9112-6f387aefaa50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b68575-298b-487d-9112-6f387aefaa50-plugins-config-volume\") pod \"f2b68575-298b-487d-9112-6f387aefaa50\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " Oct 30 01:17:14.894575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.892501 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b68575-298b-487d-9112-6f387aefaa50-kube-api-access-vz5sh\" (UniqueName: \"kubernetes.io/projected/f2b68575-298b-487d-9112-6f387aefaa50-kube-api-access-vz5sh\") pod \"f2b68575-298b-487d-9112-6f387aefaa50\" (UID: \"f2b68575-298b-487d-9112-6f387aefaa50\") " Oct 30 01:17:14.894963 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.894915 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b68575-298b-487d-9112-6f387aefaa50-plugins-config-volume" pod "f2b68575-298b-487d-9112-6f387aefaa50" (UID: "f2b68575-298b-487d-9112-6f387aefaa50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:14.897484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.897463 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b68575-298b-487d-9112-6f387aefaa50-kube-api-access-vz5sh" pod "f2b68575-298b-487d-9112-6f387aefaa50" (UID: "f2b68575-298b-487d-9112-6f387aefaa50"). InnerVolumeSpecName "kube-api-access-vz5sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:14.995404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.995324 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b68575-298b-487d-9112-6f387aefaa50-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:14.995404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:14.995357 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5sh\" (UniqueName: \"kubernetes.io/projected/f2b68575-298b-487d-9112-6f387aefaa50-kube-api-access-vz5sh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.070935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.070688 4905 generic.go:356] "Generic (PLEG): container finished" podID="8144991b-f4ea-4314-962e-376ff644a7d3" containerID="1f15df15dcdde245046b19abceac281e0df119d150902ed05f00f31cfbb4139e" exitCode=0 Oct 30 01:17:15.070935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.070803 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" event={"ID":"8144991b-f4ea-4314-962e-376ff644a7d3","Type":"ContainerDied","Data":"1f15df15dcdde245046b19abceac281e0df119d150902ed05f00f31cfbb4139e"} Oct 30 01:17:15.072558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.072499 4905 generic.go:356] "Generic (PLEG): container finished" podID="f2b68575-298b-487d-9112-6f387aefaa50" containerID="d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828" exitCode=0 Oct 30 01:17:15.074619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.072846 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" event={"ID":"f2b68575-298b-487d-9112-6f387aefaa50","Type":"ContainerDied","Data":"d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828"} Oct 30 01:17:15.074619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.072897 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" event={"ID":"f2b68575-298b-487d-9112-6f387aefaa50","Type":"ContainerDied","Data":"43305f581e1a4032bc556afa5a2a1d97b42a9b46badefaaca8040968d3ad3c7c"} Oct 30 01:17:15.074619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.072926 4905 scope.go:122] "RemoveContainer" containerID="d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828" Oct 30 01:17:15.074619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.073083 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj" Oct 30 01:17:15.081297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.081239 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" event={"ID":"a8674f5b-07a1-4d8b-abee-a37cca9206ad","Type":"ContainerStarted","Data":"a7ad9e475f9ad77bed160b7bd1150c17fefb8c93a48b14e051855d721ced898d"} Oct 30 01:17:15.093935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.093917 4905 scope.go:122] "RemoveContainer" containerID="d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828" Oct 30 01:17:15.094157 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:17:15.094139 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828\": container with ID starting with d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828 not found: ID does not exist" containerID="d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828" Oct 30 01:17:15.094240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.094166 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828"} err="failed to get container status \"d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828\": rpc error: code = NotFound desc = could not find container \"d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828\": container with ID starting with d4b31f59cd9f14608abf86f22737688b979765c794824cb6184291ea65426828 not found: ID does not exist" Oct 30 01:17:15.202071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.202046 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:17:15.209869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.209839 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj"] Oct 30 01:17:15.210088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.210075 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95fe33-8957f5756-r6jzj"] Oct 30 01:17:15.300203 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300174 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-model-cache\") pod \"8144991b-f4ea-4314-962e-376ff644a7d3\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " Oct 30 01:17:15.300376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300270 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-home\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-home\") pod \"8144991b-f4ea-4314-962e-376ff644a7d3\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " Oct 30 01:17:15.300376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300342 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-dshm\") pod \"8144991b-f4ea-4314-962e-376ff644a7d3\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " Oct 30 01:17:15.300494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300431 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8144991b-f4ea-4314-962e-376ff644a7d3-kube-api-access-mj9jb\" (UniqueName: \"kubernetes.io/projected/8144991b-f4ea-4314-962e-376ff644a7d3-kube-api-access-mj9jb\") pod \"8144991b-f4ea-4314-962e-376ff644a7d3\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " Oct 30 01:17:15.300494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300474 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8144991b-f4ea-4314-962e-376ff644a7d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8144991b-f4ea-4314-962e-376ff644a7d3-tls-certs\") pod \"8144991b-f4ea-4314-962e-376ff644a7d3\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " Oct 30 01:17:15.300638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300468 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-model-cache" pod "8144991b-f4ea-4314-962e-376ff644a7d3" (UID: "8144991b-f4ea-4314-962e-376ff644a7d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.300638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300596 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-home" pod "8144991b-f4ea-4314-962e-376ff644a7d3" (UID: "8144991b-f4ea-4314-962e-376ff644a7d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.300638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.300613 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-tmp-dir\") pod \"8144991b-f4ea-4314-962e-376ff644a7d3\" (UID: \"8144991b-f4ea-4314-962e-376ff644a7d3\") " Oct 30 01:17:15.301053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.301030 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-tmp-dir" pod "8144991b-f4ea-4314-962e-376ff644a7d3" (UID: "8144991b-f4ea-4314-962e-376ff644a7d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.301373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.301353 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.301373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.301375 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.301579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.301383 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.303268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.303238 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8144991b-f4ea-4314-962e-376ff644a7d3-kube-api-access-mj9jb" pod "8144991b-f4ea-4314-962e-376ff644a7d3" (UID: "8144991b-f4ea-4314-962e-376ff644a7d3"). InnerVolumeSpecName "kube-api-access-mj9jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:15.303367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.303309 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8144991b-f4ea-4314-962e-376ff644a7d3-tls-certs" pod "8144991b-f4ea-4314-962e-376ff644a7d3" (UID: "8144991b-f4ea-4314-962e-376ff644a7d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:15.303367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.303309 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-dshm" pod "8144991b-f4ea-4314-962e-376ff644a7d3" (UID: "8144991b-f4ea-4314-962e-376ff644a7d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.368016 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:17:15.367892 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38c63d8_1547_4dff_ba87_1adf7d32c7d6.slice/crio-15bace2264561d31dda3872d564098641e7bfe12974b708252e4564018ba014f WatchSource:0}: Error finding container 15bace2264561d31dda3872d564098641e7bfe12974b708252e4564018ba014f: Status 404 returned error can't find the container with id 15bace2264561d31dda3872d564098641e7bfe12974b708252e4564018ba014f Oct 30 01:17:15.368388 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.368359 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx"] Oct 30 01:17:15.402543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.402491 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8144991b-f4ea-4314-962e-376ff644a7d3-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.402644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.402552 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj9jb\" (UniqueName: \"kubernetes.io/projected/8144991b-f4ea-4314-962e-376ff644a7d3-kube-api-access-mj9jb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.402644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.402565 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8144991b-f4ea-4314-962e-376ff644a7d3-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.460739 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.460707 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf"] Oct 30 01:17:15.470941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.470899 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:15.471027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.470996 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:15.471092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.471042 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:15.957771 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:15.957674 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" podUID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:17:16.123893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.123862 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" Oct 30 01:17:16.123893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.123876 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74" event={"ID":"8144991b-f4ea-4314-962e-376ff644a7d3","Type":"ContainerDied","Data":"40aca10e05dec995e1bffad15cdcc491248ed6029aa02a8eb26bb6fe50eaf7df"} Oct 30 01:17:16.124230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.123917 4905 scope.go:122] "RemoveContainer" containerID="1f15df15dcdde245046b19abceac281e0df119d150902ed05f00f31cfbb4139e" Oct 30 01:17:16.129918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.128995 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" event={"ID":"a38c63d8-1547-4dff-ba87-1adf7d32c7d6","Type":"ContainerStarted","Data":"ede17193dd60076c4baff7969fae73b8f0fe2a2706d27bcf32f2e45857234320"} Oct 30 01:17:16.129918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.129049 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" event={"ID":"a38c63d8-1547-4dff-ba87-1adf7d32c7d6","Type":"ContainerStarted","Data":"15bace2264561d31dda3872d564098641e7bfe12974b708252e4564018ba014f"} Oct 30 01:17:16.129918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.129854 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:16.134796 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.134770 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" event={"ID":"0c6a1e07-822c-443b-bf43-293a1455bc8b","Type":"ContainerStarted","Data":"9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496"} Oct 30 01:17:16.134929 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.134819 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" event={"ID":"0c6a1e07-822c-443b-bf43-293a1455bc8b","Type":"ContainerStarted","Data":"b9f9043a1f0059c6ef801b40fb29a9018668f4fd5b021496840777c7de852d25"} Oct 30 01:17:16.141906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.141882 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" event={"ID":"a8674f5b-07a1-4d8b-abee-a37cca9206ad","Type":"ContainerStarted","Data":"c1c90627815a972c33aeb1a3505500d5ed97882e4e5c133c46195f22da4af353"} Oct 30 01:17:16.142024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.141957 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:16.185315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.185283 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74"] Oct 30 01:17:16.191035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.191006 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca324d/sub-test-model-ca324d-kserve-75fdf44b7-ssq74"] Oct 30 01:17:16.232403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.232349 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podStartSLOduration=2.232330933 podStartE2EDuration="2.232330933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:16.227162119 +0000 UTC m=+2321.170335313" watchObservedRunningTime="2026-10-30 01:17:16.232330933 +0000 UTC m=+2321.175504126" Oct 30 01:17:16.272306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.272261 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" podStartSLOduration=3.272247554 podStartE2EDuration="3.272247554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:16.268420434 +0000 UTC m=+2321.211593624" watchObservedRunningTime="2026-10-30 01:17:16.272247554 +0000 UTC m=+2321.215420764" Oct 30 01:17:16.312883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.312833 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" podStartSLOduration=3.312819228 podStartE2EDuration="3.312819228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:16.308968744 +0000 UTC m=+2321.252141935" watchObservedRunningTime="2026-10-30 01:17:16.312819228 +0000 UTC m=+2321.255992421" Oct 30 01:17:16.814681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.814651 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:16.820451 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:16.820427 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:17.155063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:17.154959 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:17.155988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:17.155966 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:17:17.194201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:17.194171 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8144991b-f4ea-4314-962e-376ff644a7d3" path="/var/lib/kubelet/pods/8144991b-f4ea-4314-962e-376ff644a7d3/volumes" Oct 30 01:17:17.194677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:17.194657 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b68575-298b-487d-9112-6f387aefaa50" path="/var/lib/kubelet/pods/f2b68575-298b-487d-9112-6f387aefaa50/volumes" Oct 30 01:17:20.192091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.192005 4905 generic.go:356] "Generic (PLEG): container finished" podID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerID="8251a58ab9944e283df7ed3eb80d2d678d314344ce1baa56d14ce1953d3de845" exitCode=0 Oct 30 01:17:20.192091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.192074 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" event={"ID":"e5e381e2-5b6b-49af-8fe2-5ba37a189955","Type":"ContainerDied","Data":"8251a58ab9944e283df7ed3eb80d2d678d314344ce1baa56d14ce1953d3de845"} Oct 30 01:17:20.192885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.192482 4905 scope.go:122] "RemoveContainer" containerID="8251a58ab9944e283df7ed3eb80d2d678d314344ce1baa56d14ce1953d3de845" Oct 30 01:17:20.194492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.194458 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:20.194627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.194547 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:20.194627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.194580 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:20.399345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.399306 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4"] Oct 30 01:17:20.400806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.400779 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8144991b-f4ea-4314-962e-376ff644a7d3" containerName="main" Oct 30 01:17:20.400806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.400807 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8144991b-f4ea-4314-962e-376ff644a7d3" containerName="main" Oct 30 01:17:20.400974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.400831 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f2b68575-298b-487d-9112-6f387aefaa50" containerName="payload-processing" Oct 30 01:17:20.400974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.400840 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b68575-298b-487d-9112-6f387aefaa50" containerName="payload-processing" Oct 30 01:17:20.401063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.401021 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8144991b-f4ea-4314-962e-376ff644a7d3" containerName="main" Oct 30 01:17:20.401063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.401037 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b68575-298b-487d-9112-6f387aefaa50" containerName="payload-processing" Oct 30 01:17:20.405525 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.405495 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:20.421502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.421458 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1075cb\"" Oct 30 01:17:20.421673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.421585 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1075cb-dockercfg-rp45w\"" Oct 30 01:17:20.422710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.422688 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4"] Oct 30 01:17:20.552541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.545805 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27767101-8a0d-4f03-9f35-4097a9cac1e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:20.552541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.546015 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56r8m\" (UniqueName: \"kubernetes.io/projected/27767101-8a0d-4f03-9f35-4097a9cac1e1-kube-api-access-56r8m\") pod \"payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:20.678553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.652612 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56r8m\" (UniqueName: \"kubernetes.io/projected/27767101-8a0d-4f03-9f35-4097a9cac1e1-kube-api-access-56r8m\") pod \"payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:20.678553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.652850 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27767101-8a0d-4f03-9f35-4097a9cac1e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:20.678553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.653828 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27767101-8a0d-4f03-9f35-4097a9cac1e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:20.746841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:20.746715 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56r8m\" (UniqueName: \"kubernetes.io/projected/27767101-8a0d-4f03-9f35-4097a9cac1e1-kube-api-access-56r8m\") pod \"payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:21.027631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:21.027602 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:21.202102 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:21.202029 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" event={"ID":"e5e381e2-5b6b-49af-8fe2-5ba37a189955","Type":"ContainerStarted","Data":"606cc8fdd96b924ebda96667ec9e976411a15db2e74a5760892b5e778d8db0c7"} Oct 30 01:17:21.357077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:21.357044 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4"] Oct 30 01:17:21.360408 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:17:21.360382 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27767101_8a0d_4f03_9f35_4097a9cac1e1.slice/crio-42a10f2de4b3cbe431118934f7b9c9c331c435fe36ad226762df9bbd16d53b28 WatchSource:0}: Error finding container 42a10f2de4b3cbe431118934f7b9c9c331c435fe36ad226762df9bbd16d53b28: Status 404 returned error can't find the container with id 42a10f2de4b3cbe431118934f7b9c9c331c435fe36ad226762df9bbd16d53b28 Oct 30 01:17:22.061737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:22.060936 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:17:22.062672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:22.062643 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 01:17:22.062828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:22.062700 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 01:17:22.211413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:22.211371 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" event={"ID":"27767101-8a0d-4f03-9f35-4097a9cac1e1","Type":"ContainerStarted","Data":"fa673839fef5e95aa754f27d2106ed02177f0edfcace1315adabc54210ba551b"} Oct 30 01:17:22.211860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:22.211438 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" event={"ID":"27767101-8a0d-4f03-9f35-4097a9cac1e1","Type":"ContainerStarted","Data":"42a10f2de4b3cbe431118934f7b9c9c331c435fe36ad226762df9bbd16d53b28"} Oct 30 01:17:22.211860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:22.211466 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:22.257910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:22.257852 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" podStartSLOduration=2.257836232 podStartE2EDuration="2.257836232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:22.255938532 +0000 UTC m=+2327.199111735" watchObservedRunningTime="2026-10-30 01:17:22.257836232 +0000 UTC m=+2327.201009467" Oct 30 01:17:23.057545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:23.057417 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 01:17:23.057545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:23.057480 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 01:17:24.057502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:24.057471 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 01:17:24.058039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:24.057541 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 01:17:25.059879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:25.059842 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 01:17:25.060461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:25.060433 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 01:17:25.955924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:25.955880 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" podUID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:17:26.057730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:26.057694 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 01:17:26.058660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:26.058619 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 01:17:26.759083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:26.759049 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:17:27.063502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:27.063421 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:17:27.157969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:27.157938 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:17:27.283651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:27.283625 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:17:28.174577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:28.174439 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:17:33.222111 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:33.222081 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:17:34.014396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.014371 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:17:34.126696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.126662 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8c09b6-109b-4b35-9789-5ad9b705efe6-kube-api-access-9l4wh\" (UniqueName: \"kubernetes.io/projected/1b8c09b6-109b-4b35-9789-5ad9b705efe6-kube-api-access-9l4wh\") pod \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " Oct 30 01:17:34.126881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.126811 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8c09b6-109b-4b35-9789-5ad9b705efe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c09b6-109b-4b35-9789-5ad9b705efe6-plugins-config-volume\") pod \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\" (UID: \"1b8c09b6-109b-4b35-9789-5ad9b705efe6\") " Oct 30 01:17:34.127174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.127149 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8c09b6-109b-4b35-9789-5ad9b705efe6-plugins-config-volume" pod "1b8c09b6-109b-4b35-9789-5ad9b705efe6" (UID: "1b8c09b6-109b-4b35-9789-5ad9b705efe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:34.127351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.127336 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c09b6-109b-4b35-9789-5ad9b705efe6-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:34.128927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.128907 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c09b6-109b-4b35-9789-5ad9b705efe6-kube-api-access-9l4wh" pod "1b8c09b6-109b-4b35-9789-5ad9b705efe6" (UID: "1b8c09b6-109b-4b35-9789-5ad9b705efe6"). InnerVolumeSpecName "kube-api-access-9l4wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:34.229046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.229004 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4wh\" (UniqueName: \"kubernetes.io/projected/1b8c09b6-109b-4b35-9789-5ad9b705efe6-kube-api-access-9l4wh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:34.440230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.440139 4905 generic.go:356] "Generic (PLEG): container finished" podID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" containerID="90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43" exitCode=0 Oct 30 01:17:34.440230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.440187 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" Oct 30 01:17:34.440466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.440228 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" event={"ID":"1b8c09b6-109b-4b35-9789-5ad9b705efe6","Type":"ContainerDied","Data":"90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43"} Oct 30 01:17:34.440466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.440279 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv" event={"ID":"1b8c09b6-109b-4b35-9789-5ad9b705efe6","Type":"ContainerDied","Data":"fb7b1a67db06e75e294bafe832e3dc953127529690cae1aa0c9e86670783c8b5"} Oct 30 01:17:34.440466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.440304 4905 scope.go:122] "RemoveContainer" containerID="90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43" Oct 30 01:17:34.456688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.456654 4905 scope.go:122] "RemoveContainer" containerID="90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43" Oct 30 01:17:34.457006 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:17:34.456984 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43\": container with ID starting with 90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43 not found: ID does not exist" containerID="90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43" Oct 30 01:17:34.457108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.457016 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43"} err="failed to get container status \"90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43\": rpc error: code = NotFound desc = could not find container \"90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43\": container with ID starting with 90e8ea8fccaa276135915119aad1ffb8025f3ec1b47713524e3545267705ac43 not found: ID does not exist" Oct 30 01:17:34.497982 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.497952 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv"] Oct 30 01:17:34.511669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:34.511642 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95fe33-b5884d87f-knxzv"] Oct 30 01:17:35.193757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.193727 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" path="/var/lib/kubelet/pods/1b8c09b6-109b-4b35-9789-5ad9b705efe6/volumes" Oct 30 01:17:35.738429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.738388 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x"] Oct 30 01:17:35.739356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.739336 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" containerName="payload-pre-processing" Oct 30 01:17:35.739444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.739357 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" containerName="payload-pre-processing" Oct 30 01:17:35.739503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.739493 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8c09b6-109b-4b35-9789-5ad9b705efe6" containerName="payload-pre-processing" Oct 30 01:17:35.744572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.744554 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.766307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.766277 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.766307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.766284 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x"] Oct 30 01:17:35.766625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.766486 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.766683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.766646 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e37aa37-d267-4038-93ff-6510ffd77977-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.766834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.766809 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.766940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.766920 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zb8c\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-kube-api-access-6zb8c\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.767033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.767013 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e37aa37-d267-4038-93ff-6510ffd77977-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.767165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.767106 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.767165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.767149 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.767293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.767246 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.868860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.868827 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e37aa37-d267-4038-93ff-6510ffd77977-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.868935 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.868984 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zb8c\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-kube-api-access-6zb8c\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869040 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e37aa37-d267-4038-93ff-6510ffd77977-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869199 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869262 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869311 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869323 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869417 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869465 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869575 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e37aa37-d267-4038-93ff-6510ffd77977-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869643 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869766 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.869963 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.869943 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.871694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.871668 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.871805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.871676 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e37aa37-d267-4038-93ff-6510ffd77977-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.891032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.891003 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zb8c\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-kube-api-access-6zb8c\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:35.891673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:35.891649 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:36.059680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:36.059598 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:36.384119 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:17:36.384091 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e37aa37_d267_4038_93ff_6510ffd77977.slice/crio-b81e9b81dc5b42015ce10ed15024f347635a43f61d213da4abd547b16c89fcca WatchSource:0}: Error finding container b81e9b81dc5b42015ce10ed15024f347635a43f61d213da4abd547b16c89fcca: Status 404 returned error can't find the container with id b81e9b81dc5b42015ce10ed15024f347635a43f61d213da4abd547b16c89fcca Oct 30 01:17:36.384452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:36.384429 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x"] Oct 30 01:17:36.387019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:36.386985 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:36.387109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:36.387050 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:36.387109 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:36.387079 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:36.460357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:36.460320 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" event={"ID":"8e37aa37-d267-4038-93ff-6510ffd77977","Type":"ContainerStarted","Data":"b81e9b81dc5b42015ce10ed15024f347635a43f61d213da4abd547b16c89fcca"} Oct 30 01:17:37.471715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:37.471660 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" event={"ID":"8e37aa37-d267-4038-93ff-6510ffd77977","Type":"ContainerStarted","Data":"7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93"} Oct 30 01:17:37.559670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:37.559581 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" podStartSLOduration=2.559558037 podStartE2EDuration="2.559558037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:37.550875114 +0000 UTC m=+2342.494048288" watchObservedRunningTime="2026-10-30 01:17:37.559558037 +0000 UTC m=+2342.502731227" Oct 30 01:17:38.060370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:38.060338 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:38.061688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:38.061667 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:17:38.061761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:38.061713 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 01:17:38.986199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:38.986161 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh"] Oct 30 01:17:38.986727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:38.986556 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" containerID="cri-o://75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5" gracePeriod=30 Oct 30 01:17:39.060327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:39.060289 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:17:39.060548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:39.060360 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 01:17:40.066935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.066898 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:17:40.067342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.066978 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 01:17:40.175624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.175591 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq"] Oct 30 01:17:40.183556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.182295 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.210978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.210946 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq"] Oct 30 01:17:40.212581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.212561 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-60f38f-openshift-default-dockercfg-tvpjg\"" Oct 30 01:17:40.319046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.318952 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sf42\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-kube-api-access-5sf42\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319073 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319107 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319255 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319320 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319408 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319625 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319667 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.319722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.319703 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.422869 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.422930 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.422978 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.423029 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sf42\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-kube-api-access-5sf42\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.423064 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.423092 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.423246 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.423330 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.424564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.423393 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.442474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.441938 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.442474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.442161 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.444604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.444569 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.444984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.444967 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.445462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.445442 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.454820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.454793 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.461286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.461263 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.474665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.474609 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sf42\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-kube-api-access-5sf42\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.557989 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.557924 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:40.811013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:40.810979 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:41.072080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.071836 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:17:41.072711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.072113 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 01:17:41.173656 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:17:41.173628 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c229f1_7e63_41f2_977a_043fa4ca9c26.slice/crio-953666ba881ffe78b9db99e50c49942a4516ab797ed78a0a40ccfdaaabf9ef17 WatchSource:0}: Error finding container 953666ba881ffe78b9db99e50c49942a4516ab797ed78a0a40ccfdaaabf9ef17: Status 404 returned error can't find the container with id 953666ba881ffe78b9db99e50c49942a4516ab797ed78a0a40ccfdaaabf9ef17 Oct 30 01:17:41.176021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.175995 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq"] Oct 30 01:17:41.176542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.176493 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:41.176636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.176606 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:41.176700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.176654 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:17:41.574319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.574277 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" event={"ID":"b0c229f1-7e63-41f2-977a-043fa4ca9c26","Type":"ContainerStarted","Data":"d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0"} Oct 30 01:17:41.574527 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.574357 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" event={"ID":"b0c229f1-7e63-41f2-977a-043fa4ca9c26","Type":"ContainerStarted","Data":"953666ba881ffe78b9db99e50c49942a4516ab797ed78a0a40ccfdaaabf9ef17"} Oct 30 01:17:41.619775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.619725 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podStartSLOduration=1.619710695 podStartE2EDuration="1.619710695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:41.61822447 +0000 UTC m=+2346.561397663" watchObservedRunningTime="2026-10-30 01:17:41.619710695 +0000 UTC m=+2346.562883887" Oct 30 01:17:41.817545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:41.815751 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:42.061885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:42.061851 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:17:42.062126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:42.062085 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 01:17:42.460736 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:17:42.460650 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:17:42.820503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:42.820459 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:43.066083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:43.066041 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:43.066573 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:43.066513 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:43.067826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:43.067802 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:17:43.660650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:43.660619 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:43.661710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:43.661691 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:17:44.448545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.445998 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512198 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-credential-socket\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512273 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istiod-ca-cert\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512333 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-envoy\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512388 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-socket\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512584 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-certs\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512634 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-podinfo\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512726 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-data\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512769 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-kube-api-access-tdtxz\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-kube-api-access-tdtxz\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.512814 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-token\") pod \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\" (UID: \"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485\") " Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.514373 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-socket" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.514595 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-certs" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.514779 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-data" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.515050 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istiod-ca-cert" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.515546 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-credential-socket" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.521554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.521008 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-kube-api-access-tdtxz" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "kube-api-access-tdtxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:44.571022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.563472 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-envoy" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.571747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.571715 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-token" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:44.572203 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.572167 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-podinfo" pod "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" (UID: "5f4c4e40-abc2-4612-9b8e-fa2c3ba67485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:44.614644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614619 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614641 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614654 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614664 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614672 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614681 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614690 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614698 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdtxz\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-kube-api-access-tdtxz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.614835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.614706 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.672933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.672901 4905 generic.go:356] "Generic (PLEG): container finished" podID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerID="75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5" exitCode=0 Oct 30 01:17:44.673407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.672961 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" Oct 30 01:17:44.673407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.672998 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" event={"ID":"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485","Type":"ContainerDied","Data":"75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5"} Oct 30 01:17:44.673407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.673063 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh" event={"ID":"5f4c4e40-abc2-4612-9b8e-fa2c3ba67485","Type":"ContainerDied","Data":"817557aaed213972f3dbe4f06a46d1ae9cd90788c117c0bb0c5b9418e59cf1c8"} Oct 30 01:17:44.673407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.673093 4905 scope.go:122] "RemoveContainer" containerID="75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5" Oct 30 01:17:44.684895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.684873 4905 scope.go:122] "RemoveContainer" containerID="6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4" Oct 30 01:17:44.695263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.695247 4905 scope.go:122] "RemoveContainer" containerID="75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5" Oct 30 01:17:44.695511 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:17:44.695495 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5\": container with ID starting with 75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5 not found: ID does not exist" containerID="75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5" Oct 30 01:17:44.695511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.695530 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5"} err="failed to get container status \"75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5\": rpc error: code = NotFound desc = could not find container \"75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5\": container with ID starting with 75f15a1dcc8d140158fb85a0a791ae802dbfac59109d2a46cce59ab0527ac4c5 not found: ID does not exist" Oct 30 01:17:44.695662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.695546 4905 scope.go:122] "RemoveContainer" containerID="6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4" Oct 30 01:17:44.695805 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:17:44.695777 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4\": container with ID starting with 6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4 not found: ID does not exist" containerID="6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4" Oct 30 01:17:44.695856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.695812 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4"} err="failed to get container status \"6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4\": rpc error: code = NotFound desc = could not find container \"6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4\": container with ID starting with 6c3a14aefa243bf057f4d61ee2ee8896cf5ac2d3ed8c72b8c38b4042966b61b4 not found: ID does not exist" Oct 30 01:17:44.733476 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.733447 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh"] Oct 30 01:17:44.737424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:44.737399 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-kfgjh"] Oct 30 01:17:45.077548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:45.077484 4905 scope.go:122] "RemoveContainer" containerID="4341babe456c8ab8a344c715035ed66a45665d5c3691fdabc13f8d3bb06407d3" Oct 30 01:17:45.193512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:45.193485 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" path="/var/lib/kubelet/pods/5f4c4e40-abc2-4612-9b8e-fa2c3ba67485/volumes" Oct 30 01:17:46.078197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.078165 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk"] Oct 30 01:17:46.079014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.078998 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" Oct 30 01:17:46.079071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.079016 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" Oct 30 01:17:46.079071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.079034 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" Oct 30 01:17:46.079071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.079039 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" Oct 30 01:17:46.079205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.079195 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" Oct 30 01:17:46.079240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.079208 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4c4e40-abc2-4612-9b8e-fa2c3ba67485" containerName="istio-proxy" Oct 30 01:17:46.084968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.084947 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.118051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.118011 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk"] Oct 30 01:17:46.132877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.118361 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-60f38f\"" Oct 30 01:17:46.132877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.118809 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-60f38f-dockercfg-psgff\"" Oct 30 01:17:46.288220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.288170 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211cc2b7-9c6a-4213-b954-51bea62aa9b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.288417 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.288381 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/211cc2b7-9c6a-4213-b954-51bea62aa9b1-kube-api-access-twfm2\") pod \"payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.440660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.418259 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211cc2b7-9c6a-4213-b954-51bea62aa9b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.440660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.418538 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/211cc2b7-9c6a-4213-b954-51bea62aa9b1-kube-api-access-twfm2\") pod \"payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.440660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.419870 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211cc2b7-9c6a-4213-b954-51bea62aa9b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.443691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.442649 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk"] Oct 30 01:17:46.454352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.454305 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:46.499319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.499264 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk"] Oct 30 01:17:46.506744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.505725 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/211cc2b7-9c6a-4213-b954-51bea62aa9b1-kube-api-access-twfm2\") pod \"payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.523265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.523231 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f928f4d2-b9fc-4ad7-954f-6cc37c140272-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:46.523412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.523343 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tls\" (UniqueName: \"kubernetes.io/projected/f928f4d2-b9fc-4ad7-954f-6cc37c140272-kube-api-access-89tls\") pod \"payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:46.625492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.625460 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f928f4d2-b9fc-4ad7-954f-6cc37c140272-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:46.625697 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.625648 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89tls\" (UniqueName: \"kubernetes.io/projected/f928f4d2-b9fc-4ad7-954f-6cc37c140272-kube-api-access-89tls\") pod \"payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:46.626152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.626131 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f928f4d2-b9fc-4ad7-954f-6cc37c140272-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:46.648159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.648129 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tls\" (UniqueName: \"kubernetes.io/projected/f928f4d2-b9fc-4ad7-954f-6cc37c140272-kube-api-access-89tls\") pod \"payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:46.711470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.711395 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:46.823142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:46.823112 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:47.022663 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:17:47.022635 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211cc2b7_9c6a_4213_b954_51bea62aa9b1.slice/crio-0566c59764855c8d34cd883c7e2301497d939f712d4a79e64595560bce3c5060 WatchSource:0}: Error finding container 0566c59764855c8d34cd883c7e2301497d939f712d4a79e64595560bce3c5060: Status 404 returned error can't find the container with id 0566c59764855c8d34cd883c7e2301497d939f712d4a79e64595560bce3c5060 Oct 30 01:17:47.024179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.024157 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk"] Oct 30 01:17:47.137099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.136784 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk"] Oct 30 01:17:47.697397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.697358 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" event={"ID":"f928f4d2-b9fc-4ad7-954f-6cc37c140272","Type":"ContainerStarted","Data":"859296d7de1cec95e06d6ffe2d9ae43bbf40e0f246f8b600ab302f8a4ea7a55f"} Oct 30 01:17:47.697628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.697428 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" event={"ID":"f928f4d2-b9fc-4ad7-954f-6cc37c140272","Type":"ContainerStarted","Data":"c23d110b4729133e79e20e64c727f97d0e5bb5d1075e398b56d90e3906d1f929"} Oct 30 01:17:47.697628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.697591 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:47.699320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.699293 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" event={"ID":"211cc2b7-9c6a-4213-b954-51bea62aa9b1","Type":"ContainerStarted","Data":"a491d5fce012bb888087a39d0268a06647b0215677b33a696cf7396297882fba"} Oct 30 01:17:47.699442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.699342 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" event={"ID":"211cc2b7-9c6a-4213-b954-51bea62aa9b1","Type":"ContainerStarted","Data":"0566c59764855c8d34cd883c7e2301497d939f712d4a79e64595560bce3c5060"} Oct 30 01:17:47.699442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.699363 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:17:47.752556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.752490 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" podStartSLOduration=1.752473701 podStartE2EDuration="1.752473701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:47.737884537 +0000 UTC m=+2352.681057732" watchObservedRunningTime="2026-10-30 01:17:47.752473701 +0000 UTC m=+2352.695646893" Oct 30 01:17:47.780236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:47.780192 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" podStartSLOduration=1.780181304 podStartE2EDuration="1.780181304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:47.777291849 +0000 UTC m=+2352.720465042" watchObservedRunningTime="2026-10-30 01:17:47.780181304 +0000 UTC m=+2352.723354494" Oct 30 01:17:58.710583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:58.710549 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:17:58.711713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:17:58.711686 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:18:12.415722 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:18:12.415683 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:18:19.711309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.711233 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf"] Oct 30 01:18:19.717347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.717326 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.740065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.740037 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf"] Oct 30 01:18:19.759780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.759749 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-60f38f-serving-cert\"" Oct 30 01:18:19.815978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.815946 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.816151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.815983 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd667\" (UniqueName: \"kubernetes.io/projected/36ec974d-9690-40be-b7ae-a8839b79d6db-kube-api-access-cd667\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.816151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.816022 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.816253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.816159 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec974d-9690-40be-b7ae-a8839b79d6db-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.939209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.939154 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.949118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.949075 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd667\" (UniqueName: \"kubernetes.io/projected/36ec974d-9690-40be-b7ae-a8839b79d6db-kube-api-access-cd667\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.950969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.950933 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.952670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.952646 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec974d-9690-40be-b7ae-a8839b79d6db-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.954097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.953795 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec974d-9690-40be-b7ae-a8839b79d6db-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.970588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.969115 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.970588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.970148 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:19.972362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:19.972335 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd667\" (UniqueName: \"kubernetes.io/projected/36ec974d-9690-40be-b7ae-a8839b79d6db-kube-api-access-cd667\") pod \"maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:20.043349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:20.043319 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:20.351976 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:18:20.351946 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ec974d_9690_40be_b7ae_a8839b79d6db.slice/crio-69b1e9532d4e01da0460ac1222193314a96c266f643eb56f7e5ef419f2973ffe WatchSource:0}: Error finding container 69b1e9532d4e01da0460ac1222193314a96c266f643eb56f7e5ef419f2973ffe: Status 404 returned error can't find the container with id 69b1e9532d4e01da0460ac1222193314a96c266f643eb56f7e5ef419f2973ffe Oct 30 01:18:20.354819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:20.354792 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf"] Oct 30 01:18:21.056307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:21.056262 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" event={"ID":"36ec974d-9690-40be-b7ae-a8839b79d6db","Type":"ContainerStarted","Data":"0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355"} Oct 30 01:18:21.056852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:21.056320 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" event={"ID":"36ec974d-9690-40be-b7ae-a8839b79d6db","Type":"ContainerStarted","Data":"69b1e9532d4e01da0460ac1222193314a96c266f643eb56f7e5ef419f2973ffe"} Oct 30 01:18:21.056852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:21.056349 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:21.118186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:21.118137 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" podStartSLOduration=1.81711507 podStartE2EDuration="2.118122655s" totalImagesPullingTime="301.007585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:21.09730229 +0000 UTC m=+2386.040475483" watchObservedRunningTime="2026-10-30 01:18:21.118122655 +0000 UTC m=+2386.061295846" Oct 30 01:18:27.073952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:27.073919 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:18:27.808434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:27.808407 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt"] Oct 30 01:18:27.844646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:27.844605 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx"] Oct 30 01:18:32.827256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:32.819634 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" podUID="a8674f5b-07a1-4d8b-abee-a37cca9206ad" containerName="payload-processing" containerID="cri-o://c1c90627815a972c33aeb1a3505500d5ed97882e4e5c133c46195f22da4af353" gracePeriod=25 Oct 30 01:18:32.852485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:32.850488 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" podUID="a38c63d8-1547-4dff-ba87-1adf7d32c7d6" containerName="payload-pre-processing" containerID="cri-o://ede17193dd60076c4baff7969fae73b8f0fe2a2706d27bcf32f2e45857234320" gracePeriod=25 Oct 30 01:18:33.211600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.210686 4905 generic.go:356] "Generic (PLEG): container finished" podID="a38c63d8-1547-4dff-ba87-1adf7d32c7d6" containerID="ede17193dd60076c4baff7969fae73b8f0fe2a2706d27bcf32f2e45857234320" exitCode=0 Oct 30 01:18:33.211600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.210820 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" event={"ID":"a38c63d8-1547-4dff-ba87-1adf7d32c7d6","Type":"ContainerDied","Data":"ede17193dd60076c4baff7969fae73b8f0fe2a2706d27bcf32f2e45857234320"} Oct 30 01:18:33.214373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.213181 4905 generic.go:356] "Generic (PLEG): container finished" podID="a8674f5b-07a1-4d8b-abee-a37cca9206ad" containerID="c1c90627815a972c33aeb1a3505500d5ed97882e4e5c133c46195f22da4af353" exitCode=0 Oct 30 01:18:33.214373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.213383 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" event={"ID":"a8674f5b-07a1-4d8b-abee-a37cca9206ad","Type":"ContainerDied","Data":"c1c90627815a972c33aeb1a3505500d5ed97882e4e5c133c46195f22da4af353"} Oct 30 01:18:33.597209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.597176 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:18:33.619274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.619219 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8674f5b-07a1-4d8b-abee-a37cca9206ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8674f5b-07a1-4d8b-abee-a37cca9206ad-plugins-config-volume\") pod \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " Oct 30 01:18:33.619710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.619683 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8674f5b-07a1-4d8b-abee-a37cca9206ad-plugins-config-volume" pod "a8674f5b-07a1-4d8b-abee-a37cca9206ad" (UID: "a8674f5b-07a1-4d8b-abee-a37cca9206ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:33.621484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.621461 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8674f5b-07a1-4d8b-abee-a37cca9206ad-kube-api-access-859gl\" (UniqueName: \"kubernetes.io/projected/a8674f5b-07a1-4d8b-abee-a37cca9206ad-kube-api-access-859gl\") pod \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\" (UID: \"a8674f5b-07a1-4d8b-abee-a37cca9206ad\") " Oct 30 01:18:33.622263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.622239 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8674f5b-07a1-4d8b-abee-a37cca9206ad-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:18:33.623947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.623922 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8674f5b-07a1-4d8b-abee-a37cca9206ad-kube-api-access-859gl" pod "a8674f5b-07a1-4d8b-abee-a37cca9206ad" (UID: "a8674f5b-07a1-4d8b-abee-a37cca9206ad"). InnerVolumeSpecName "kube-api-access-859gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:33.724211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.724180 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859gl\" (UniqueName: \"kubernetes.io/projected/a8674f5b-07a1-4d8b-abee-a37cca9206ad-kube-api-access-859gl\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:18:33.772298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.772220 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:18:33.848544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.835373 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-kube-api-access-xqd2j\" (UniqueName: \"kubernetes.io/projected/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-kube-api-access-xqd2j\") pod \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " Oct 30 01:18:33.848544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.835455 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-plugins-config-volume\") pod \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\" (UID: \"a38c63d8-1547-4dff-ba87-1adf7d32c7d6\") " Oct 30 01:18:33.848544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.846200 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-kube-api-access-xqd2j" pod "a38c63d8-1547-4dff-ba87-1adf7d32c7d6" (UID: "a38c63d8-1547-4dff-ba87-1adf7d32c7d6"). InnerVolumeSpecName "kube-api-access-xqd2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:33.873542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.865774 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-plugins-config-volume" pod "a38c63d8-1547-4dff-ba87-1adf7d32c7d6" (UID: "a38c63d8-1547-4dff-ba87-1adf7d32c7d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:33.941543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.941068 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqd2j\" (UniqueName: \"kubernetes.io/projected/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-kube-api-access-xqd2j\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:18:33.941543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:33.941103 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c63d8-1547-4dff-ba87-1adf7d32c7d6-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.222344 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.222268 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" Oct 30 01:18:34.222546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.222299 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx" event={"ID":"a38c63d8-1547-4dff-ba87-1adf7d32c7d6","Type":"ContainerDied","Data":"15bace2264561d31dda3872d564098641e7bfe12974b708252e4564018ba014f"} Oct 30 01:18:34.222718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.222702 4905 scope.go:122] "RemoveContainer" containerID="ede17193dd60076c4baff7969fae73b8f0fe2a2706d27bcf32f2e45857234320" Oct 30 01:18:34.223944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.223916 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" event={"ID":"a8674f5b-07a1-4d8b-abee-a37cca9206ad","Type":"ContainerDied","Data":"a7ad9e475f9ad77bed160b7bd1150c17fefb8c93a48b14e051855d721ced898d"} Oct 30 01:18:34.224048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.223932 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt" Oct 30 01:18:34.289763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.289739 4905 scope.go:122] "RemoveContainer" containerID="c1c90627815a972c33aeb1a3505500d5ed97882e4e5c133c46195f22da4af353" Oct 30 01:18:34.323046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.323018 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt"] Oct 30 01:18:34.338544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.338492 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f461c3-6fbc886db6-kqfgt"] Oct 30 01:18:34.390544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.389571 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx"] Oct 30 01:18:34.395621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:34.395443 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f461c3-5bcfb9bd99-l8bkx"] Oct 30 01:18:35.200763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:35.200734 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38c63d8-1547-4dff-ba87-1adf7d32c7d6" path="/var/lib/kubelet/pods/a38c63d8-1547-4dff-ba87-1adf7d32c7d6/volumes" Oct 30 01:18:35.201761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:35.201742 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8674f5b-07a1-4d8b-abee-a37cca9206ad" path="/var/lib/kubelet/pods/a8674f5b-07a1-4d8b-abee-a37cca9206ad/volumes" Oct 30 01:18:35.219254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:35.219234 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4"] Oct 30 01:18:36.378544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:36.376151 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:18:36.414101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:36.414073 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:18:40.223618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.222038 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" podUID="27767101-8a0d-4f03-9f35-4097a9cac1e1" containerName="payload-pre-processing" containerID="cri-o://fa673839fef5e95aa754f27d2106ed02177f0edfcace1315adabc54210ba551b" gracePeriod=25 Oct 30 01:18:40.403182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.403140 4905 generic.go:356] "Generic (PLEG): container finished" podID="27767101-8a0d-4f03-9f35-4097a9cac1e1" containerID="fa673839fef5e95aa754f27d2106ed02177f0edfcace1315adabc54210ba551b" exitCode=0 Oct 30 01:18:40.403368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.403229 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" event={"ID":"27767101-8a0d-4f03-9f35-4097a9cac1e1","Type":"ContainerDied","Data":"fa673839fef5e95aa754f27d2106ed02177f0edfcace1315adabc54210ba551b"} Oct 30 01:18:40.576144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.576113 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:18:40.629975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.629939 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27767101-8a0d-4f03-9f35-4097a9cac1e1-kube-api-access-56r8m\" (UniqueName: \"kubernetes.io/projected/27767101-8a0d-4f03-9f35-4097a9cac1e1-kube-api-access-56r8m\") pod \"27767101-8a0d-4f03-9f35-4097a9cac1e1\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " Oct 30 01:18:40.632574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.630294 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27767101-8a0d-4f03-9f35-4097a9cac1e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27767101-8a0d-4f03-9f35-4097a9cac1e1-plugins-config-volume\") pod \"27767101-8a0d-4f03-9f35-4097a9cac1e1\" (UID: \"27767101-8a0d-4f03-9f35-4097a9cac1e1\") " Oct 30 01:18:40.632574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.630800 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27767101-8a0d-4f03-9f35-4097a9cac1e1-plugins-config-volume" pod "27767101-8a0d-4f03-9f35-4097a9cac1e1" (UID: "27767101-8a0d-4f03-9f35-4097a9cac1e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:40.632574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.631478 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27767101-8a0d-4f03-9f35-4097a9cac1e1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:18:40.633867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.633828 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27767101-8a0d-4f03-9f35-4097a9cac1e1-kube-api-access-56r8m" pod "27767101-8a0d-4f03-9f35-4097a9cac1e1" (UID: "27767101-8a0d-4f03-9f35-4097a9cac1e1"). InnerVolumeSpecName "kube-api-access-56r8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:40.732912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:40.732880 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56r8m\" (UniqueName: \"kubernetes.io/projected/27767101-8a0d-4f03-9f35-4097a9cac1e1-kube-api-access-56r8m\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:18:41.414825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:41.414792 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" Oct 30 01:18:41.414825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:41.414813 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4" event={"ID":"27767101-8a0d-4f03-9f35-4097a9cac1e1","Type":"ContainerDied","Data":"42a10f2de4b3cbe431118934f7b9c9c331c435fe36ad226762df9bbd16d53b28"} Oct 30 01:18:41.415358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:41.414885 4905 scope.go:122] "RemoveContainer" containerID="fa673839fef5e95aa754f27d2106ed02177f0edfcace1315adabc54210ba551b" Oct 30 01:18:41.474645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:41.474563 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4"] Oct 30 01:18:41.487994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:41.487958 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1075cb-565c685cdc-pk7m4"] Oct 30 01:18:41.759675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:41.759646 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:18:41.759863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:41.759691 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:18:42.440413 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:18:42.440373 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:18:43.193763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:43.193731 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27767101-8a0d-4f03-9f35-4097a9cac1e1" path="/var/lib/kubelet/pods/27767101-8a0d-4f03-9f35-4097a9cac1e1/volumes" Oct 30 01:18:43.436856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:43.436826 4905 generic.go:356] "Generic (PLEG): container finished" podID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerID="9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496" exitCode=0 Oct 30 01:18:43.437083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:43.436903 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" event={"ID":"0c6a1e07-822c-443b-bf43-293a1455bc8b","Type":"ContainerDied","Data":"9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496"} Oct 30 01:18:43.437401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:43.437366 4905 scope.go:122] "RemoveContainer" containerID="9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496" Oct 30 01:18:43.439354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:43.439324 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:43.439448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:43.439395 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:43.439448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:43.439425 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:44.462102 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.462066 4905 generic.go:356] "Generic (PLEG): container finished" podID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerID="a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a" exitCode=0 Oct 30 01:18:44.462696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.462129 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" event={"ID":"52df6f9a-b866-4fa6-8b50-16606cb8e1ac","Type":"ContainerDied","Data":"a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a"} Oct 30 01:18:44.462782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.462714 4905 scope.go:122] "RemoveContainer" containerID="a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a" Oct 30 01:18:44.464784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.464750 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:44.464884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.464785 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" event={"ID":"0c6a1e07-822c-443b-bf43-293a1455bc8b","Type":"ContainerStarted","Data":"8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06"} Oct 30 01:18:44.464884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.464823 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:44.464884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.464856 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:44.814385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.814348 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:18:44.816073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.816044 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 01:18:44.816194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:44.816111 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 01:18:45.496918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:45.496881 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" event={"ID":"52df6f9a-b866-4fa6-8b50-16606cb8e1ac","Type":"ContainerStarted","Data":"e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186"} Oct 30 01:18:45.814066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:45.813979 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 01:18:45.814066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:45.814044 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 01:18:46.017373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:46.017346 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:18:46.017632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:46.017409 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:18:46.017780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:46.017762 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 01:18:46.814694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:46.814659 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 01:18:46.815160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:46.814717 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 01:18:47.017316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.017279 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:18:47.017558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.017342 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 01:18:47.185598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.178632 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:18:47.314977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.314945 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5"] Oct 30 01:18:47.315769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315754 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a38c63d8-1547-4dff-ba87-1adf7d32c7d6" containerName="payload-pre-processing" Oct 30 01:18:47.315769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315770 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38c63d8-1547-4dff-ba87-1adf7d32c7d6" containerName="payload-pre-processing" Oct 30 01:18:47.315769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315801 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a8674f5b-07a1-4d8b-abee-a37cca9206ad" containerName="payload-processing" Oct 30 01:18:47.315970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315807 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8674f5b-07a1-4d8b-abee-a37cca9206ad" containerName="payload-processing" Oct 30 01:18:47.315970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315815 4905 cpu_manager.go:414] "containerMap: removing container" podUID="27767101-8a0d-4f03-9f35-4097a9cac1e1" containerName="payload-pre-processing" Oct 30 01:18:47.315970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315820 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27767101-8a0d-4f03-9f35-4097a9cac1e1" containerName="payload-pre-processing" Oct 30 01:18:47.315970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315958 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38c63d8-1547-4dff-ba87-1adf7d32c7d6" containerName="payload-pre-processing" Oct 30 01:18:47.315970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315970 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8674f5b-07a1-4d8b-abee-a37cca9206ad" containerName="payload-processing" Oct 30 01:18:47.316226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.315977 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="27767101-8a0d-4f03-9f35-4097a9cac1e1" containerName="payload-pre-processing" Oct 30 01:18:47.319342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.319320 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:47.360545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.348837 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5c515e\"" Oct 30 01:18:47.384357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.384326 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5c515e-dockercfg-ssrfw\"" Oct 30 01:18:47.420607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.400627 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:47.421030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.420999 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76q7\" (UniqueName: \"kubernetes.io/projected/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-kube-api-access-n76q7\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:47.480618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.477985 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5"] Oct 30 01:18:47.829474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.829422 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:47.853901 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.853733 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76q7\" (UniqueName: \"kubernetes.io/projected/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-kube-api-access-n76q7\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:47.856412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.842636 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 01:18:47.861877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.861845 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 01:18:47.863329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.855040 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:47.983605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:47.980485 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76q7\" (UniqueName: \"kubernetes.io/projected/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-kube-api-access-n76q7\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:48.017379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.017351 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:18:48.017572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.017397 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 01:18:48.093180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.092105 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:48.503214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.503181 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5"] Oct 30 01:18:48.503608 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:18:48.503581 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1472f130_50a7_4ff6_a8aa_448b4dbfd59b.slice/crio-050774372efc561d745ab2e10a44a40d4034c9c339be302a9661e5f3f44c8724 WatchSource:0}: Error finding container 050774372efc561d745ab2e10a44a40d4034c9c339be302a9661e5f3f44c8724: Status 404 returned error can't find the container with id 050774372efc561d745ab2e10a44a40d4034c9c339be302a9661e5f3f44c8724 Oct 30 01:18:48.814572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.814514 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 01:18:48.814700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.814594 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 01:18:48.992377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.991250 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" event={"ID":"1472f130-50a7-4ff6-a8aa-448b4dbfd59b","Type":"ContainerStarted","Data":"1faaf01a4c2c49c1bb6a62dd32bb1d255119ed33cac86b7e834ccde743d9483b"} Oct 30 01:18:48.992377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.991313 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" event={"ID":"1472f130-50a7-4ff6-a8aa-448b4dbfd59b","Type":"ContainerStarted","Data":"050774372efc561d745ab2e10a44a40d4034c9c339be302a9661e5f3f44c8724"} Oct 30 01:18:48.992377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:48.991571 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:18:49.019545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:49.016880 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:18:49.019545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:49.016935 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 01:18:49.046587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:49.039047 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" podStartSLOduration=2.039026448 podStartE2EDuration="2.039026448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:49.035963869 +0000 UTC m=+2413.979137062" watchObservedRunningTime="2026-10-30 01:18:49.039026448 +0000 UTC m=+2413.982199636" Oct 30 01:18:49.835273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:49.835237 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 01:18:49.835629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:49.835596 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 01:18:50.016933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:50.016897 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:18:50.017319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:50.016953 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 01:18:50.818811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:50.818778 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:18:50.819975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:50.819952 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:18:51.022149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:51.022118 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:18:51.022645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:51.022553 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:18:51.023233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:51.023211 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:18:56.953253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:56.952938 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr"] Oct 30 01:18:56.960949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:56.960919 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:56.979639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:56.979607 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-cb31e3-openshift-default-dockercfg-cgb6r\"" Oct 30 01:18:57.000670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.000637 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr"] Oct 30 01:18:57.087711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.087678 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-certs\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.087866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.087731 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-credential-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.087866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.087832 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-podinfo\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.087945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.087871 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.087945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.087897 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrsv8\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-kube-api-access-hrsv8\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.088049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.088037 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-data\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.088085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.088060 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istiod-ca-cert\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.088085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.088077 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-envoy\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.088152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.088096 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-token\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189390 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-certs\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189440 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-credential-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189510 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-podinfo\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189581 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189620 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrsv8\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-kube-api-access-hrsv8\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189775 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-data\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189811 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istiod-ca-cert\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189821 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-certs\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189839 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-envoy\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.189898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.189880 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-token\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.190161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.190076 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-data\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.190296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.190269 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.190500 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.190479 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-credential-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.190843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.190818 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istiod-ca-cert\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.192335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.192312 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-podinfo\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.192458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.192440 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-envoy\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.210446 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.210356 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-token\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.210446 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.210381 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrsv8\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-kube-api-access-hrsv8\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.276962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.276937 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:57.593832 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:18:57.593801 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b712d86_c47a_41a9_9a25_1ae45c0ba963.slice/crio-d38a5ef0ab942cf49fed41210089e368d370a4ce6563c1b6afd5f820986cce5c WatchSource:0}: Error finding container d38a5ef0ab942cf49fed41210089e368d370a4ce6563c1b6afd5f820986cce5c: Status 404 returned error can't find the container with id d38a5ef0ab942cf49fed41210089e368d370a4ce6563c1b6afd5f820986cce5c Oct 30 01:18:57.596059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.596032 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr"] Oct 30 01:18:57.596846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.596815 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:57.596912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.596881 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:57.596912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:57.596907 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:18:58.072871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:58.072830 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" event={"ID":"2b712d86-c47a-41a9-9a25-1ae45c0ba963","Type":"ContainerStarted","Data":"3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65"} Oct 30 01:18:58.073324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:58.072901 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" event={"ID":"2b712d86-c47a-41a9-9a25-1ae45c0ba963","Type":"ContainerStarted","Data":"d38a5ef0ab942cf49fed41210089e368d370a4ce6563c1b6afd5f820986cce5c"} Oct 30 01:18:58.119037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:58.118991 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podStartSLOduration=2.118977787 podStartE2EDuration="2.118977787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:58.117056505 +0000 UTC m=+2423.060229696" watchObservedRunningTime="2026-10-30 01:18:58.118977787 +0000 UTC m=+2423.062151036" Oct 30 01:18:58.277858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:58.277820 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:18:58.279397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:58.279371 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 30 01:18:58.279567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:58.279426 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 30 01:18:59.277589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:59.277554 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 30 01:18:59.278028 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:18:59.277607 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 30 01:19:00.003379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:00.003341 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:19:00.278349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:00.278264 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 30 01:19:00.278349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:00.278310 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 30 01:19:01.278072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:01.278035 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 30 01:19:01.278257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:01.278121 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 30 01:19:02.283545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:02.282729 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 30 01:19:02.283545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:02.282813 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 30 01:19:03.283562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:03.279912 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 30 01:19:03.283562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:03.279985 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 30 01:19:03.961547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:03.954323 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:04.204543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:04.185988 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:04.309859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:04.309642 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 30 01:19:04.309859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:04.309703 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 30 01:19:05.286423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:05.286388 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:19:05.286974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:05.286951 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:19:05.289583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:05.288805 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:19:10.465962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.465924 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27"] Oct 30 01:19:10.473645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.473445 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.508348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.508317 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27"] Oct 30 01:19:10.588896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.588866 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.588925 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.589051 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.589076 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f736e26-d57f-48d1-992a-971e4d67c65e-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.589099 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.589222 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jcx\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-kube-api-access-x4jcx\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.589289 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.589379 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.589545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.589508 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.690964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.690930 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jcx\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-kube-api-access-x4jcx\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691009 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691047 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691162 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691208 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691241 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691294 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691319 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f736e26-d57f-48d1-992a-971e4d67c65e-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691326 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691340 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691624 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.691767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691692 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.692015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.691988 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.692282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.692245 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f736e26-d57f-48d1-992a-971e4d67c65e-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.693811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.693790 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.694023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.694004 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.714745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.714717 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.715247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.715229 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jcx\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-kube-api-access-x4jcx\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:10.811513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:10.811492 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:11.134888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:11.134860 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27"] Oct 30 01:19:11.137843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:11.137801 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:11.137952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:11.137897 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:11.138002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:11.137946 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:11.670543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:11.670265 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" event={"ID":"3f736e26-d57f-48d1-992a-971e4d67c65e","Type":"ContainerStarted","Data":"1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8"} Oct 30 01:19:11.670543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:11.670326 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" event={"ID":"3f736e26-d57f-48d1-992a-971e4d67c65e","Type":"ContainerStarted","Data":"a15fb9a2e23b15dbb7df19391aaac214f803d0c12eb23d5432463ce66367c72d"} Oct 30 01:19:11.812000 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:11.811964 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:12.447542 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:19:12.444199 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:19:12.458633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.457358 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podStartSLOduration=2.457335918 podStartE2EDuration="2.457335918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:11.737058754 +0000 UTC m=+2436.680231946" watchObservedRunningTime="2026-10-30 01:19:12.457335918 +0000 UTC m=+2437.400509114" Oct 30 01:19:12.459333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.459292 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7"] Oct 30 01:19:12.467954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.467932 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:12.485230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.483864 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7"] Oct 30 01:19:12.485230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.484089 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-954bf6\"" Oct 30 01:19:12.485230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.484089 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-954bf6-dockercfg-jlk2s\"" Oct 30 01:19:12.638820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.610983 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9g6\" (UniqueName: \"kubernetes.io/projected/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-kube-api-access-qv9g6\") pod \"payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:12.638820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.611126 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:12.783845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.783802 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9g6\" (UniqueName: \"kubernetes.io/projected/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-kube-api-access-qv9g6\") pod \"payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:12.809042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.809006 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:12.822672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.814622 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5"] Oct 30 01:19:12.833812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.833763 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:12.843496 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.843463 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 30 01:19:12.844044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.844011 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 30 01:19:12.858003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.857973 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5"] Oct 30 01:19:12.858290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.858275 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:12.895035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.895003 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9g6\" (UniqueName: \"kubernetes.io/projected/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-kube-api-access-qv9g6\") pod \"payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:12.933543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.924576 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2h8t\" (UniqueName: \"kubernetes.io/projected/1ff91ee3-305c-4562-a05b-4eac7828ef1c-kube-api-access-z2h8t\") pod \"payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:12.934108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:12.934081 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff91ee3-305c-4562-a05b-4eac7828ef1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:13.048763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.048661 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2h8t\" (UniqueName: \"kubernetes.io/projected/1ff91ee3-305c-4562-a05b-4eac7828ef1c-kube-api-access-z2h8t\") pod \"payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:13.050953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.050927 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff91ee3-305c-4562-a05b-4eac7828ef1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:13.065253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.065217 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff91ee3-305c-4562-a05b-4eac7828ef1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:13.080180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.080150 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2h8t\" (UniqueName: \"kubernetes.io/projected/1ff91ee3-305c-4562-a05b-4eac7828ef1c-kube-api-access-z2h8t\") pod \"payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:13.094773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.094744 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:13.179561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.179487 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:13.485411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.483455 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7"] Oct 30 01:19:13.487068 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:19:13.487033 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd0d9fb_f75a_4f0a_b984_1f64ad207014.slice/crio-869c4d131830f36be01bbd7f22f2d34d0d6d016855f46f97d1a982a8ef1df0dc WatchSource:0}: Error finding container 869c4d131830f36be01bbd7f22f2d34d0d6d016855f46f97d1a982a8ef1df0dc: Status 404 returned error can't find the container with id 869c4d131830f36be01bbd7f22f2d34d0d6d016855f46f97d1a982a8ef1df0dc Oct 30 01:19:13.563434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.563356 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5"] Oct 30 01:19:13.574224 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:19:13.574200 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff91ee3_305c_4562_a05b_4eac7828ef1c.slice/crio-9fc541c03ac576eccb21094e38c5109a10ddda5e38a2290a83d8998639e49eb0 WatchSource:0}: Error finding container 9fc541c03ac576eccb21094e38c5109a10ddda5e38a2290a83d8998639e49eb0: Status 404 returned error can't find the container with id 9fc541c03ac576eccb21094e38c5109a10ddda5e38a2290a83d8998639e49eb0 Oct 30 01:19:13.812402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.812365 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 30 01:19:13.812885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.812417 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 30 01:19:13.916923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.916846 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" event={"ID":"1ff91ee3-305c-4562-a05b-4eac7828ef1c","Type":"ContainerStarted","Data":"056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0"} Oct 30 01:19:13.916923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.916897 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" event={"ID":"1ff91ee3-305c-4562-a05b-4eac7828ef1c","Type":"ContainerStarted","Data":"9fc541c03ac576eccb21094e38c5109a10ddda5e38a2290a83d8998639e49eb0"} Oct 30 01:19:13.916923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.916924 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:13.918830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.918799 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" event={"ID":"6fd0d9fb-f75a-4f0a-b984-1f64ad207014","Type":"ContainerStarted","Data":"9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4"} Oct 30 01:19:13.918957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.918861 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" event={"ID":"6fd0d9fb-f75a-4f0a-b984-1f64ad207014","Type":"ContainerStarted","Data":"869c4d131830f36be01bbd7f22f2d34d0d6d016855f46f97d1a982a8ef1df0dc"} Oct 30 01:19:13.918957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.918894 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:13.966543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:13.964705 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" podStartSLOduration=1.964687192 podStartE2EDuration="1.964687192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:13.962892515 +0000 UTC m=+2438.906065719" watchObservedRunningTime="2026-10-30 01:19:13.964687192 +0000 UTC m=+2438.907860386" Oct 30 01:19:14.007692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.007639 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" podStartSLOduration=2.007626139 podStartE2EDuration="2.007626139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:14.003075711 +0000 UTC m=+2438.946248904" watchObservedRunningTime="2026-10-30 01:19:14.007626139 +0000 UTC m=+2438.950799330" Oct 30 01:19:14.716637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.716610 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:19:14.719399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.716667 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:14.811794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.811756 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 30 01:19:14.811995 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.811806 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 30 01:19:14.938702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.936945 4905 generic.go:356] "Generic (PLEG): container finished" podID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerID="d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0" exitCode=0 Oct 30 01:19:14.938702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.938018 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" event={"ID":"b0c229f1-7e63-41f2-977a-043fa4ca9c26","Type":"ContainerDied","Data":"d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0"} Oct 30 01:19:14.938702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.938452 4905 scope.go:122] "RemoveContainer" containerID="d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0" Oct 30 01:19:14.950562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.949927 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:14.950562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.950025 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:14.950562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:14.950071 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:15.823542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.822829 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 30 01:19:15.823542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.822886 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 30 01:19:15.823542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.823314 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc"] Oct 30 01:19:15.882010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.876893 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:15.883719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.883690 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e213edcc-ae82-4ab7-8220-81965e6e6c52-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:15.884206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.884177 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdf7\" (UniqueName: \"kubernetes.io/projected/e213edcc-ae82-4ab7-8220-81965e6e6c52-kube-api-access-wmdf7\") pod \"payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:15.888455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.888283 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc"] Oct 30 01:19:15.899581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.899206 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1075cb-dockercfg-rh4dd\"" Oct 30 01:19:15.899581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.899324 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1075cb\"" Oct 30 01:19:15.990827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.990590 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdf7\" (UniqueName: \"kubernetes.io/projected/e213edcc-ae82-4ab7-8220-81965e6e6c52-kube-api-access-wmdf7\") pod \"payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:15.990827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.990745 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e213edcc-ae82-4ab7-8220-81965e6e6c52-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:15.995373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:15.995339 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e213edcc-ae82-4ab7-8220-81965e6e6c52-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:16.010193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.004110 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" event={"ID":"b0c229f1-7e63-41f2-977a-043fa4ca9c26","Type":"ContainerStarted","Data":"1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f"} Oct 30 01:19:16.045553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.044905 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdf7\" (UniqueName: \"kubernetes.io/projected/e213edcc-ae82-4ab7-8220-81965e6e6c52-kube-api-access-wmdf7\") pod \"payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:16.204589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.204504 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:16.615337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.605613 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc"] Oct 30 01:19:16.812305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.812275 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 30 01:19:16.812475 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.812322 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 30 01:19:16.812812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.812788 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:19:16.812950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.812822 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 01:19:16.812950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:16.812852 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 01:19:17.016281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:17.016234 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" event={"ID":"e213edcc-ae82-4ab7-8220-81965e6e6c52","Type":"ContainerStarted","Data":"88c813dfe499794f7d93f04fa8be2a4a2168a21b19b57b4617e6365aa3530607"} Oct 30 01:19:17.812391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:17.812358 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 30 01:19:17.812608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:17.812407 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 30 01:19:17.812608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:17.812590 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 01:19:17.812712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:17.812623 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 01:19:18.811884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:18.811852 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 01:19:18.812354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:18.811898 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 01:19:18.818832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:18.818805 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:18.820198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:18.820137 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:18.820616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:18.820597 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:19:19.811992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:19.811958 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 01:19:19.812449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:19.812018 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 01:19:20.811932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:20.811902 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 01:19:20.812104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:20.811951 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 01:19:21.194643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:21.194566 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:21.194643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:21.194622 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" event={"ID":"e213edcc-ae82-4ab7-8220-81965e6e6c52","Type":"ContainerStarted","Data":"0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d"} Oct 30 01:19:21.242616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:21.242508 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" podStartSLOduration=2.812357175 podStartE2EDuration="6.242487299s" totalImagesPullingTime="3.430130124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:21.230930191 +0000 UTC m=+2446.174103387" watchObservedRunningTime="2026-10-30 01:19:21.242487299 +0000 UTC m=+2446.185660492" Oct 30 01:19:21.811750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:21.811721 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 01:19:21.811930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:21.811777 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 01:19:22.817224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:22.817195 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:19:22.817728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:22.817706 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:19:22.818488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:22.818469 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:19:24.941660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:24.941633 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:19:24.949499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:24.949474 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:19:32.344444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:32.344409 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:19:38.554831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.554794 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm"] Oct 30 01:19:38.564548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.562084 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.597801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.597771 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5c5a66-openshift-default-dockercfg-rq8j5\"" Oct 30 01:19:38.598321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.598290 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm"] Oct 30 01:19:38.771543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.770937 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-certs\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771094 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9mm\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-kube-api-access-fg9mm\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771334 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-credential-socket\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771398 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-data\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771453 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-podinfo\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771607 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-socket\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771640 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-token\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771664 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-envoy\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.771971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.771795 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaf339a-6d1f-460e-8b17-57b3494d6707-istiod-ca-cert\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.878637 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-socket\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.878684 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-token\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.878715 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-envoy\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.878798 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaf339a-6d1f-460e-8b17-57b3494d6707-istiod-ca-cert\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.878835 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-certs\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.878927 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9mm\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-kube-api-access-fg9mm\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.879081 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-credential-socket\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.879120 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-data\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.879164 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-podinfo\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.880487 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaf339a-6d1f-460e-8b17-57b3494d6707-istiod-ca-cert\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.880815 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-socket\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.884025 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-certs\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.884163 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-data\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.887592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.886233 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-credential-socket\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.893560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.893505 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-podinfo\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.914154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.907267 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-token\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.935844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.922613 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9mm\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-kube-api-access-fg9mm\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:38.936914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:38.936620 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-envoy\") pod \"e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:39.206365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:39.206273 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:39.580971 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:19:39.580942 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaf339a_6d1f_460e_8b17_57b3494d6707.slice/crio-3d90c84f210c2a54ef61322c7b970f1fb9b00fe36ce782171a240807b665190a WatchSource:0}: Error finding container 3d90c84f210c2a54ef61322c7b970f1fb9b00fe36ce782171a240807b665190a: Status 404 returned error can't find the container with id 3d90c84f210c2a54ef61322c7b970f1fb9b00fe36ce782171a240807b665190a Oct 30 01:19:39.584244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:39.584207 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:39.584366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:39.584305 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:39.584366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:39.584343 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:39.591281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:39.591258 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm"] Oct 30 01:19:40.488631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.488597 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" event={"ID":"8aaf339a-6d1f-460e-8b17-57b3494d6707","Type":"ContainerStarted","Data":"179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141"} Oct 30 01:19:40.488831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.488656 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" event={"ID":"8aaf339a-6d1f-460e-8b17-57b3494d6707","Type":"ContainerStarted","Data":"3d90c84f210c2a54ef61322c7b970f1fb9b00fe36ce782171a240807b665190a"} Oct 30 01:19:40.565004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.564958 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podStartSLOduration=2.564945068 podStartE2EDuration="2.564945068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:40.56121535 +0000 UTC m=+2465.504388542" watchObservedRunningTime="2026-10-30 01:19:40.564945068 +0000 UTC m=+2465.508118259" Oct 30 01:19:40.566971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.566019 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6"] Oct 30 01:19:40.577569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.572415 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.596030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.596006 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6"] Oct 30 01:19:40.621468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621437 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.621657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621488 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqljg\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-kube-api-access-gqljg\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.621734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621677 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.621801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621764 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.621801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621792 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.621907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621867 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.621962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621909 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.622023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.621968 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.622090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.622072 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725030 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725097 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725172 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725278 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725384 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725455 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqljg\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-kube-api-access-gqljg\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725649 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725726 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.725753 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.727948 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.728204 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.728224 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.728249 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.731107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.728444 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.732936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.732906 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.733111 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.733089 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.753841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.753781 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.754118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.754099 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqljg\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-kube-api-access-gqljg\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:40.904545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:40.903691 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:41.206870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.206804 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:41.214243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.214222 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:41.246275 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:19:41.246246 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ade1c6_0236_43a4_8e65_3644cb096c5c.slice/crio-614e4012cd032b5d19f63f5255d46b6b0e61edfd2a51561feea67b09759488a6 WatchSource:0}: Error finding container 614e4012cd032b5d19f63f5255d46b6b0e61edfd2a51561feea67b09759488a6: Status 404 returned error can't find the container with id 614e4012cd032b5d19f63f5255d46b6b0e61edfd2a51561feea67b09759488a6 Oct 30 01:19:41.246756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.246735 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6"] Oct 30 01:19:41.249889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.249856 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:41.249978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.249935 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:41.250021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.249978 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:19:41.498613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.498578 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" event={"ID":"e3ade1c6-0236-43a4-8e65-3644cb096c5c","Type":"ContainerStarted","Data":"ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246"} Oct 30 01:19:41.498825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.498635 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" event={"ID":"e3ade1c6-0236-43a4-8e65-3644cb096c5c","Type":"ContainerStarted","Data":"614e4012cd032b5d19f63f5255d46b6b0e61edfd2a51561feea67b09759488a6"} Oct 30 01:19:41.499085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.499063 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:41.500071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.500049 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:19:41.546554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.544997 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podStartSLOduration=1.544980435 podStartE2EDuration="1.544980435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:41.543658592 +0000 UTC m=+2466.486831785" watchObservedRunningTime="2026-10-30 01:19:41.544980435 +0000 UTC m=+2466.488153627" Oct 30 01:19:41.904709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:41.904624 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:42.594279 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:19:42.594214 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:19:42.905770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:42.905692 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:19:42.906181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:42.905772 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 01:19:43.904361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:43.904320 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:19:43.904608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:43.904380 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 01:19:44.904979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:44.904485 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:19:44.904979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:44.904566 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 01:19:45.266545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.265812 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd"] Oct 30 01:19:45.273062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.273039 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:45.282957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.282918 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd"] Oct 30 01:19:45.292778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.292750 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c5a66-dockercfg-xnnxf\"" Oct 30 01:19:45.293369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.293347 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c5a66\"" Oct 30 01:19:45.304384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.304340 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:45.304501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.304389 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9sl\" (UniqueName: \"kubernetes.io/projected/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-kube-api-access-dc9sl\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:45.383620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.383579 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh"] Oct 30 01:19:45.392693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.392666 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh"] Oct 30 01:19:45.393710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.393690 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:45.407544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.407398 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:45.407544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.407444 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9sl\" (UniqueName: \"kubernetes.io/projected/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-kube-api-access-dc9sl\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:45.409403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.407913 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dba09-fa60-4a54-a692-bd142db35f83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:45.409403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.408062 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsth\" (UniqueName: \"kubernetes.io/projected/037dba09-fa60-4a54-a692-bd142db35f83-kube-api-access-qhsth\") pod \"payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:45.409403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.409061 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:45.529594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.516826 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dba09-fa60-4a54-a692-bd142db35f83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:45.742549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.736001 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsth\" (UniqueName: \"kubernetes.io/projected/037dba09-fa60-4a54-a692-bd142db35f83-kube-api-access-qhsth\") pod \"payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:45.742549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.530732 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dba09-fa60-4a54-a692-bd142db35f83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:45.742549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.739371 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9sl\" (UniqueName: \"kubernetes.io/projected/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-kube-api-access-dc9sl\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:45.780016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.779927 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsth\" (UniqueName: \"kubernetes.io/projected/037dba09-fa60-4a54-a692-bd142db35f83-kube-api-access-qhsth\") pod \"payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:45.904323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.904290 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:19:45.904546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.904339 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 01:19:45.914095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:45.914066 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:46.019231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:46.019032 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:46.928544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:46.918246 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:19:46.928544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:46.918301 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 01:19:47.356321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.356247 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:47.667213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.667021 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:19:47.712924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.712885 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd"] Oct 30 01:19:47.716123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.716085 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:47.791542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.783584 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:47.797280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.797254 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh"] Oct 30 01:19:47.913647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.894308 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" event={"ID":"60f6e4bb-e4b5-4300-ad76-9ed24889bae3","Type":"ContainerStarted","Data":"7ec097788bd5b63baa7be33c678eab7694f59dd458e78c05d3d0270f78ea223b"} Oct 30 01:19:47.915240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.915193 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:19:47.920792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:47.915599 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 01:19:48.905225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.905186 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" event={"ID":"037dba09-fa60-4a54-a692-bd142db35f83","Type":"ContainerStarted","Data":"b9ebf3d6e2e17dd41b74ac80effd94a354418f25d89b1f7a8a8fd9d60c6d94ce"} Oct 30 01:19:48.905225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.905252 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" event={"ID":"037dba09-fa60-4a54-a692-bd142db35f83","Type":"ContainerStarted","Data":"993d2986606c7d2c067af394e3abe798a050321613517f280d3670af08289d4d"} Oct 30 01:19:48.905885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.905341 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:48.907409 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.907379 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" event={"ID":"60f6e4bb-e4b5-4300-ad76-9ed24889bae3","Type":"ContainerStarted","Data":"00b2796c7126f86e3e260198b0cb0ac72f2a8787a9b11a67864677dfe234dc64"} Oct 30 01:19:48.908064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.908044 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:48.909631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.909612 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:48.909883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.909866 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:48.910477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.910460 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:19:48.949946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:48.949895 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" podStartSLOduration=3.949878885 podStartE2EDuration="3.949878885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:48.945252158 +0000 UTC m=+2473.888425350" watchObservedRunningTime="2026-10-30 01:19:48.949878885 +0000 UTC m=+2473.893052077" Oct 30 01:19:59.919697 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:59.919665 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:19:59.920225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:59.919811 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:19:59.979385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:19:59.978251 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" podStartSLOduration=14.978231038 podStartE2EDuration="14.978231038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:49.036687821 +0000 UTC m=+2473.979861006" watchObservedRunningTime="2026-10-30 01:19:59.978231038 +0000 UTC m=+2484.921404232" Oct 30 01:20:12.416876 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:20:12.416842 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:20:17.795063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:17.795023 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5"] Oct 30 01:20:22.796653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:22.796597 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" podUID="1472f130-50a7-4ff6-a8aa-448b4dbfd59b" containerName="payload-pre-processing" containerID="cri-o://1faaf01a4c2c49c1bb6a62dd32bb1d255119ed33cac86b7e834ccde743d9483b" gracePeriod=25 Oct 30 01:20:23.444859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.444824 4905 generic.go:356] "Generic (PLEG): container finished" podID="1472f130-50a7-4ff6-a8aa-448b4dbfd59b" containerID="1faaf01a4c2c49c1bb6a62dd32bb1d255119ed33cac86b7e834ccde743d9483b" exitCode=0 Oct 30 01:20:23.445323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.445304 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" event={"ID":"1472f130-50a7-4ff6-a8aa-448b4dbfd59b","Type":"ContainerDied","Data":"1faaf01a4c2c49c1bb6a62dd32bb1d255119ed33cac86b7e834ccde743d9483b"} Oct 30 01:20:23.529981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.529914 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:20:23.585613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.585044 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-plugins-config-volume\") pod \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " Oct 30 01:20:23.585613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.585238 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-kube-api-access-n76q7\" (UniqueName: \"kubernetes.io/projected/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-kube-api-access-n76q7\") pod \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\" (UID: \"1472f130-50a7-4ff6-a8aa-448b4dbfd59b\") " Oct 30 01:20:23.586360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.586320 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-plugins-config-volume" pod "1472f130-50a7-4ff6-a8aa-448b4dbfd59b" (UID: "1472f130-50a7-4ff6-a8aa-448b4dbfd59b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:23.593564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.588597 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-kube-api-access-n76q7" pod "1472f130-50a7-4ff6-a8aa-448b4dbfd59b" (UID: "1472f130-50a7-4ff6-a8aa-448b4dbfd59b"). InnerVolumeSpecName "kube-api-access-n76q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:23.762198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.762057 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:20:23.762198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.762089 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n76q7\" (UniqueName: \"kubernetes.io/projected/1472f130-50a7-4ff6-a8aa-448b4dbfd59b-kube-api-access-n76q7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:20:23.765483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.765456 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz"] Oct 30 01:20:23.785069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:23.785011 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q"] Oct 30 01:20:24.509544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:24.499110 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" event={"ID":"1472f130-50a7-4ff6-a8aa-448b4dbfd59b","Type":"ContainerDied","Data":"050774372efc561d745ab2e10a44a40d4034c9c339be302a9661e5f3f44c8724"} Oct 30 01:20:24.509544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:24.499178 4905 scope.go:122] "RemoveContainer" containerID="1faaf01a4c2c49c1bb6a62dd32bb1d255119ed33cac86b7e834ccde743d9483b" Oct 30 01:20:24.509544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:24.499362 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5" Oct 30 01:20:24.686858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:24.686826 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5"] Oct 30 01:20:24.700217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:24.700188 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-54965d96d7-qr4w5"] Oct 30 01:20:25.192800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:25.192771 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1472f130-50a7-4ff6-a8aa-448b4dbfd59b" path="/var/lib/kubelet/pods/1472f130-50a7-4ff6-a8aa-448b4dbfd59b/volumes" Oct 30 01:20:27.132615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.132581 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh"] Oct 30 01:20:27.133449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.133431 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1472f130-50a7-4ff6-a8aa-448b4dbfd59b" containerName="payload-pre-processing" Oct 30 01:20:27.133449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.133449 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1472f130-50a7-4ff6-a8aa-448b4dbfd59b" containerName="payload-pre-processing" Oct 30 01:20:27.134118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.133898 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1472f130-50a7-4ff6-a8aa-448b4dbfd59b" containerName="payload-pre-processing" Oct 30 01:20:27.139477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.139456 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.149207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.149188 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz"] Oct 30 01:20:27.155351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.154861 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.159306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.159283 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh"] Oct 30 01:20:27.160384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.160362 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5c515e-openshift-default-dockercfg-cqchw\"" Oct 30 01:20:27.226290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.226257 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz"] Oct 30 01:20:27.236845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.236813 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.236848 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxndg\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-kube-api-access-kxndg\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.236974 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.237180 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.237291 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.237315 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.237343 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.237466 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.237565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.237511 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a911e25-7c2d-424c-8a9e-462891ac0508-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339096 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afffce59-7742-4349-abc3-0ab7e9a846a1-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.339275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339146 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339263 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339299 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339335 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339363 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.339572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339454 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339485 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a911e25-7c2d-424c-8a9e-462891ac0508-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339859 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.339965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339698 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.339965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339775 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.340046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339972 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.340046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340000 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rd9j\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-kube-api-access-6rd9j\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.340132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340119 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.340167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340148 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxndg\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-kube-api-access-kxndg\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.340210 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340188 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.340297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340261 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.340297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.339561 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.340471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340332 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.340471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340346 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a911e25-7c2d-424c-8a9e-462891ac0508-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.340471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340411 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.340684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340605 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.340764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.340745 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.342315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.342292 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.342990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.342965 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.361232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.361207 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.361232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.361225 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxndg\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-kube-api-access-kxndg\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.442565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442476 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442573 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442597 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442619 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442723 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afffce59-7742-4349-abc3-0ab7e9a846a1-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442842 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442905 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442922 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.442967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.442928 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.443112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.443002 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.443112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.443030 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rd9j\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-kube-api-access-6rd9j\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.443186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.443148 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.443406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.443385 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.443484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.443443 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afffce59-7742-4349-abc3-0ab7e9a846a1-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.445373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.445347 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.445485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.445469 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.457555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.457507 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:27.462961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.462918 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.463153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.463135 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rd9j\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-kube-api-access-6rd9j\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.494940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.494914 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:27.782150 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:20:27.782121 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a911e25_7c2d_424c_8a9e_462891ac0508.slice/crio-115be8145a789b4584f13979922a97db6b307d7173876c74d4351c0e31ac927b WatchSource:0}: Error finding container 115be8145a789b4584f13979922a97db6b307d7173876c74d4351c0e31ac927b: Status 404 returned error can't find the container with id 115be8145a789b4584f13979922a97db6b307d7173876c74d4351c0e31ac927b Oct 30 01:20:27.785186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.785158 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh"] Oct 30 01:20:27.785758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.785724 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:20:27.785877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.785807 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:20:27.785877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.785858 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:20:27.827194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.827157 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz"] Oct 30 01:20:27.829778 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:20:27.829747 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafffce59_7742_4349_abc3_0ab7e9a846a1.slice/crio-2d8be7429b5ebd753b5bda587281f652b571f1b172f42248357f6993cc4b3e28 WatchSource:0}: Error finding container 2d8be7429b5ebd753b5bda587281f652b571f1b172f42248357f6993cc4b3e28: Status 404 returned error can't find the container with id 2d8be7429b5ebd753b5bda587281f652b571f1b172f42248357f6993cc4b3e28 Oct 30 01:20:27.832628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.832590 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:20:27.832739 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.832685 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:20:27.832804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:27.832735 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:20:28.637542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:28.629687 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" event={"ID":"afffce59-7742-4349-abc3-0ab7e9a846a1","Type":"ContainerStarted","Data":"cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344"} Oct 30 01:20:28.637542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:28.629760 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" event={"ID":"afffce59-7742-4349-abc3-0ab7e9a846a1","Type":"ContainerStarted","Data":"2d8be7429b5ebd753b5bda587281f652b571f1b172f42248357f6993cc4b3e28"} Oct 30 01:20:28.644554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:28.644304 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" event={"ID":"9a911e25-7c2d-424c-8a9e-462891ac0508","Type":"ContainerStarted","Data":"a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422"} Oct 30 01:20:28.644554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:28.644369 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" event={"ID":"9a911e25-7c2d-424c-8a9e-462891ac0508","Type":"ContainerStarted","Data":"115be8145a789b4584f13979922a97db6b307d7173876c74d4351c0e31ac927b"} Oct 30 01:20:28.817399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:28.794581 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" podUID="7ca97f62-eaab-4526-8241-087fcb9ba59d" containerName="payload-pre-processing" containerID="cri-o://0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87" gracePeriod=25 Oct 30 01:20:28.817399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:28.794784 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" podUID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" containerName="payload-processing" containerID="cri-o://ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142" gracePeriod=25 Oct 30 01:20:28.881546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:28.839390 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" podStartSLOduration=1.839368144 podStartE2EDuration="1.839368144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:28.698556296 +0000 UTC m=+2513.641729488" watchObservedRunningTime="2026-10-30 01:20:28.839368144 +0000 UTC m=+2513.782541336" Oct 30 01:20:29.012789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:29.006070 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" podStartSLOduration=2.006050919 podStartE2EDuration="2.006050919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:28.957375242 +0000 UTC m=+2513.900548435" watchObservedRunningTime="2026-10-30 01:20:29.006050919 +0000 UTC m=+2513.949224117" Oct 30 01:20:29.485603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:29.462931 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:29.485603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:29.463094 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 01:20:29.485603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:29.463154 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 01:20:29.497593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:29.495775 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:29.497593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:29.496822 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 01:20:29.497593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:29.496876 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 01:20:30.461543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:30.460579 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 01:20:30.461543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:30.460651 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 01:20:30.495290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:30.495253 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 01:20:30.495463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:30.495311 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 01:20:31.457927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:31.457886 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 01:20:31.458371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:31.458331 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 01:20:31.495399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:31.495268 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 01:20:31.495399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:31.495342 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 01:20:32.469607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:32.462472 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 01:20:32.469607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:32.462561 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 01:20:32.505546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:32.497932 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 01:20:32.505546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:32.498007 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 01:20:33.462162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:33.462126 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 01:20:33.462482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:33.462453 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 01:20:33.495925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:33.495891 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 01:20:33.496098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:33.495957 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 01:20:34.463664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:34.463633 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:34.464182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:34.463967 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:34.464693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:34.464668 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:20:34.499847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:34.499825 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:34.500123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:34.500099 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:34.500899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:34.500884 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:20:37.184727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:37.184693 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" podUID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:37.191248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:37.191217 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" podUID="7ca97f62-eaab-4526-8241-087fcb9ba59d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:42.418074 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:20:42.418034 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:20:47.182965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:47.182924 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" podUID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:47.190425 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:47.190390 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" podUID="7ca97f62-eaab-4526-8241-087fcb9ba59d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:49.394643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.394598 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:20:49.484080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.484055 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:20:49.528485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.528459 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca97f62-eaab-4526-8241-087fcb9ba59d-kube-api-access-n8z5h\" (UniqueName: \"kubernetes.io/projected/7ca97f62-eaab-4526-8241-087fcb9ba59d-kube-api-access-n8z5h\") pod \"7ca97f62-eaab-4526-8241-087fcb9ba59d\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " Oct 30 01:20:49.528688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.528635 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca97f62-eaab-4526-8241-087fcb9ba59d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca97f62-eaab-4526-8241-087fcb9ba59d-plugins-config-volume\") pod \"7ca97f62-eaab-4526-8241-087fcb9ba59d\" (UID: \"7ca97f62-eaab-4526-8241-087fcb9ba59d\") " Oct 30 01:20:49.530701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.529032 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca97f62-eaab-4526-8241-087fcb9ba59d-plugins-config-volume" pod "7ca97f62-eaab-4526-8241-087fcb9ba59d" (UID: "7ca97f62-eaab-4526-8241-087fcb9ba59d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:49.530701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.530678 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca97f62-eaab-4526-8241-087fcb9ba59d-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:20:49.531260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.531236 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca97f62-eaab-4526-8241-087fcb9ba59d-kube-api-access-n8z5h" pod "7ca97f62-eaab-4526-8241-087fcb9ba59d" (UID: "7ca97f62-eaab-4526-8241-087fcb9ba59d"). InnerVolumeSpecName "kube-api-access-n8z5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:49.650651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.647194 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-plugins-config-volume\") pod \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " Oct 30 01:20:49.650651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.647496 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-kube-api-access-ljnbb\" (UniqueName: \"kubernetes.io/projected/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-kube-api-access-ljnbb\") pod \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\" (UID: \"64dd91e5-c8fb-4c54-9ba0-172b6cef3028\") " Oct 30 01:20:49.650651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.648384 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8z5h\" (UniqueName: \"kubernetes.io/projected/7ca97f62-eaab-4526-8241-087fcb9ba59d-kube-api-access-n8z5h\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:20:49.691542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.685275 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-plugins-config-volume" pod "64dd91e5-c8fb-4c54-9ba0-172b6cef3028" (UID: "64dd91e5-c8fb-4c54-9ba0-172b6cef3028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:49.743785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.733878 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-kube-api-access-ljnbb" pod "64dd91e5-c8fb-4c54-9ba0-172b6cef3028" (UID: "64dd91e5-c8fb-4c54-9ba0-172b6cef3028"). InnerVolumeSpecName "kube-api-access-ljnbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:49.771839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.771775 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnbb\" (UniqueName: \"kubernetes.io/projected/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-kube-api-access-ljnbb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:20:49.771839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.771809 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd91e5-c8fb-4c54-9ba0-172b6cef3028-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:20:49.936380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.936294 4905 generic.go:356] "Generic (PLEG): container finished" podID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" containerID="ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142" exitCode=0 Oct 30 01:20:49.936380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.936355 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" Oct 30 01:20:49.936640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.936378 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" event={"ID":"64dd91e5-c8fb-4c54-9ba0-172b6cef3028","Type":"ContainerDied","Data":"ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142"} Oct 30 01:20:49.936640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.936461 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz" event={"ID":"64dd91e5-c8fb-4c54-9ba0-172b6cef3028","Type":"ContainerDied","Data":"ad0766e69dbd6e27939d8f47a1e0ed0b9fc7f032b15ba3bbe6b81dbb079a9e0e"} Oct 30 01:20:49.936640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.936495 4905 scope.go:122] "RemoveContainer" containerID="ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142" Oct 30 01:20:49.942576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.942305 4905 generic.go:356] "Generic (PLEG): container finished" podID="7ca97f62-eaab-4526-8241-087fcb9ba59d" containerID="0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87" exitCode=0 Oct 30 01:20:49.942576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.942509 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" event={"ID":"7ca97f62-eaab-4526-8241-087fcb9ba59d","Type":"ContainerDied","Data":"0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87"} Oct 30 01:20:49.942832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.942578 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" event={"ID":"7ca97f62-eaab-4526-8241-087fcb9ba59d","Type":"ContainerDied","Data":"5a546f235fef44ab1bcd91fea5f4280f5410b0a252249688a7100a8dc14a7e41"} Oct 30 01:20:49.942832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.942667 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q" Oct 30 01:20:49.959382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.959363 4905 scope.go:122] "RemoveContainer" containerID="ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142" Oct 30 01:20:49.959735 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:20:49.959716 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142\": container with ID starting with ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142 not found: ID does not exist" containerID="ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142" Oct 30 01:20:49.959806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.959742 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142"} err="failed to get container status \"ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142\": rpc error: code = NotFound desc = could not find container \"ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142\": container with ID starting with ffad07734af9e06dd9d2aeb13ed3b3656c0dab42e987173c80d305b0c96d4142 not found: ID does not exist" Oct 30 01:20:49.959806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.959772 4905 scope.go:122] "RemoveContainer" containerID="0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87" Oct 30 01:20:49.973273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.973256 4905 scope.go:122] "RemoveContainer" containerID="0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87" Oct 30 01:20:49.973805 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:20:49.973782 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87\": container with ID starting with 0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87 not found: ID does not exist" containerID="0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87" Oct 30 01:20:49.973961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:49.973937 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87"} err="failed to get container status \"0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87\": rpc error: code = NotFound desc = could not find container \"0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87\": container with ID starting with 0db9c655c75098b7c05651436ecd21c0e5494dac43843a46204887ea8c886a87 not found: ID does not exist" Oct 30 01:20:50.067541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:50.066585 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q"] Oct 30 01:20:50.227699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:50.225579 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-283eb6-5c78c64f87-8fb2q"] Oct 30 01:20:50.279545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:50.274022 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz"] Oct 30 01:20:50.279545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:50.274083 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-283eb6-6b9c5b5976-6s5zz"] Oct 30 01:20:51.193646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:51.193616 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" path="/var/lib/kubelet/pods/64dd91e5-c8fb-4c54-9ba0-172b6cef3028/volumes" Oct 30 01:20:51.194108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:20:51.194089 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca97f62-eaab-4526-8241-087fcb9ba59d" path="/var/lib/kubelet/pods/7ca97f62-eaab-4526-8241-087fcb9ba59d/volumes" Oct 30 01:21:00.464450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.464411 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd"] Oct 30 01:21:00.465508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.465485 4905 cpu_manager.go:414] "containerMap: removing container" podUID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" containerName="payload-processing" Oct 30 01:21:00.465508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.465508 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" containerName="payload-processing" Oct 30 01:21:00.465800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.465625 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7ca97f62-eaab-4526-8241-087fcb9ba59d" containerName="payload-pre-processing" Oct 30 01:21:00.465800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.465635 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca97f62-eaab-4526-8241-087fcb9ba59d" containerName="payload-pre-processing" Oct 30 01:21:00.465919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.465864 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca97f62-eaab-4526-8241-087fcb9ba59d" containerName="payload-pre-processing" Oct 30 01:21:00.465919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.465886 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="64dd91e5-c8fb-4c54-9ba0-172b6cef3028" containerName="payload-processing" Oct 30 01:21:00.470398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.470355 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.489179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.489136 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-11fbe8-openshift-default-dockercfg-j77rc\"" Oct 30 01:21:00.506020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.505990 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd"] Oct 30 01:21:00.518783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.518751 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvwqp\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-kube-api-access-nvwqp\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.518925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.518839 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istiod-ca-cert\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.518925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.518900 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-data\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.519073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.519049 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-envoy\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.519249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.519226 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-credential-socket\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.519339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.519274 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-certs\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.519339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.519300 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-podinfo\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.519339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.519322 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-token\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.519531 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.519394 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-socket\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644047 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-certs\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644098 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-podinfo\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644123 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-token\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644204 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-socket\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644322 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwqp\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-kube-api-access-nvwqp\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644381 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istiod-ca-cert\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644439 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-data\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644558 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-envoy\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644731 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-credential-socket\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.644970 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-certs\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.645090 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-credential-socket\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.645324 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-socket\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.645856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.645812 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-data\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.663877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.646425 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istiod-ca-cert\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.773542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.747487 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-envoy\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.773542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.751165 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-podinfo\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.773542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.772312 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-token\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:00.806865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:00.806784 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwqp\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-kube-api-access-nvwqp\") pod \"e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:01.101018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:01.100706 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:01.645788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:01.645756 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd"] Oct 30 01:21:01.646674 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:01.646643 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb28b3ea_2dd1_4247_9ae2_a635bd9dd10d.slice/crio-da3f2a1815f009c2b569585ace9a8036bc549bd2aa96ae77fa9b1c807bde7adc WatchSource:0}: Error finding container da3f2a1815f009c2b569585ace9a8036bc549bd2aa96ae77fa9b1c807bde7adc: Status 404 returned error can't find the container with id da3f2a1815f009c2b569585ace9a8036bc549bd2aa96ae77fa9b1c807bde7adc Oct 30 01:21:01.650110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:01.650077 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:01.650230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:01.650144 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:01.650230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:01.650182 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:02.172396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:02.172360 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" event={"ID":"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d","Type":"ContainerStarted","Data":"9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c"} Oct 30 01:21:02.172661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:02.172451 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" event={"ID":"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d","Type":"ContainerStarted","Data":"da3f2a1815f009c2b569585ace9a8036bc549bd2aa96ae77fa9b1c807bde7adc"} Oct 30 01:21:02.248111 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:02.248058 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podStartSLOduration=2.248041981 podStartE2EDuration="2.248041981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:02.2300919 +0000 UTC m=+2547.173265083" watchObservedRunningTime="2026-10-30 01:21:02.248041981 +0000 UTC m=+2547.191215238" Oct 30 01:21:03.116865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.116042 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:03.527743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.527703 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:03.572712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.572681 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:03.659593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.655783 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:21:03.838814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.838709 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp"] Oct 30 01:21:03.923452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.923413 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:03.942158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.942126 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp"] Oct 30 01:21:03.949936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.949833 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5c515e\"" Oct 30 01:21:03.952069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:03.951044 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5c515e-dockercfg-bl296\"" Oct 30 01:21:04.044010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.043957 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq66r\" (UniqueName: \"kubernetes.io/projected/248f81ea-269e-492e-abc0-0c8194898606-kube-api-access-wq66r\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:04.044232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.044102 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248f81ea-269e-492e-abc0-0c8194898606-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:04.149600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.148828 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq66r\" (UniqueName: \"kubernetes.io/projected/248f81ea-269e-492e-abc0-0c8194898606-kube-api-access-wq66r\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:04.149600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.148968 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248f81ea-269e-492e-abc0-0c8194898606-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:04.150170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.150031 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248f81ea-269e-492e-abc0-0c8194898606-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:04.183631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.175026 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq66r\" (UniqueName: \"kubernetes.io/projected/248f81ea-269e-492e-abc0-0c8194898606-kube-api-access-wq66r\") pod \"payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:04.210710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.210678 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97"] Oct 30 01:21:04.215752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.215732 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.236279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.236247 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97"] Oct 30 01:21:04.316258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.316225 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:04.354643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.354223 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s846j\" (UniqueName: \"kubernetes.io/projected/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-kube-api-access-s846j\") pod \"payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.354643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.354429 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.456884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.456848 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s846j\" (UniqueName: \"kubernetes.io/projected/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-kube-api-access-s846j\") pod \"payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.457720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.457692 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.459787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.459757 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.478091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.478070 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s846j\" (UniqueName: \"kubernetes.io/projected/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-kube-api-access-s846j\") pod \"payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.532811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.532779 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:04.635180 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:04.635143 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248f81ea_269e_492e_abc0_0c8194898606.slice/crio-a05554c1583b21e9bbf4dea6a4e4d7130b8b6128dc7d885b8314912f78b54581 WatchSource:0}: Error finding container a05554c1583b21e9bbf4dea6a4e4d7130b8b6128dc7d885b8314912f78b54581: Status 404 returned error can't find the container with id a05554c1583b21e9bbf4dea6a4e4d7130b8b6128dc7d885b8314912f78b54581 Oct 30 01:21:04.635399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.635377 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp"] Oct 30 01:21:04.867247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:04.867104 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97"] Oct 30 01:21:05.584568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.584528 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" event={"ID":"248f81ea-269e-492e-abc0-0c8194898606","Type":"ContainerStarted","Data":"cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7"} Oct 30 01:21:05.585039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.584596 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" event={"ID":"248f81ea-269e-492e-abc0-0c8194898606","Type":"ContainerStarted","Data":"a05554c1583b21e9bbf4dea6a4e4d7130b8b6128dc7d885b8314912f78b54581"} Oct 30 01:21:05.585039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.584625 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:05.586375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.586344 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" event={"ID":"8eb0e12d-c66f-43e3-a763-8a001b1b12a0","Type":"ContainerStarted","Data":"63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c"} Oct 30 01:21:05.586563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.586397 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" event={"ID":"8eb0e12d-c66f-43e3-a763-8a001b1b12a0","Type":"ContainerStarted","Data":"6f28bf0792649b863a25d79a14b7e3d00ba6eec4222b444d9ec5e0512d0717f9"} Oct 30 01:21:05.586651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.586619 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:05.629354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.629313 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" podStartSLOduration=2.629300248 podStartE2EDuration="2.629300248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:05.625112119 +0000 UTC m=+2550.568285311" watchObservedRunningTime="2026-10-30 01:21:05.629300248 +0000 UTC m=+2550.572473453" Oct 30 01:21:05.674730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:05.674633 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" podStartSLOduration=1.674613339 podStartE2EDuration="1.674613339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:05.670241055 +0000 UTC m=+2550.613414248" watchObservedRunningTime="2026-10-30 01:21:05.674613339 +0000 UTC m=+2550.617786531" Oct 30 01:21:11.349576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.349544 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m"] Oct 30 01:21:11.355042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.355020 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.395927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.393941 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-07a3a8-openshift-default-dockercfg-xw9br\"" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.397670 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.397745 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.397794 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.397841 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.397926 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.397957 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.398171 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.398204 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmmx\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-kube-api-access-szmmx\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.414542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.398382 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.432544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.422800 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m"] Oct 30 01:21:11.506753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.500287 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.506753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.500331 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmmx\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-kube-api-access-szmmx\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.507587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.507250 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.507587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.507443 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.507587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.507503 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.507897 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.507956 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.508040 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.508073 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.509223 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.510122 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.510484 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.511087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.510787 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.512147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.512101 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.556336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.556256 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.556867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.556813 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.624690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.611222 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.666143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.639963 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmmx\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-kube-api-access-szmmx\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:11.755720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:11.755673 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:12.077146 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:12.077110 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ea456d_9106_466b_b8eb_bea896ccd4b4.slice/crio-b8aeac9037c00c48c6c9f723efb6dfdddcb1c17cbbf81f51489367b0bd561a36 WatchSource:0}: Error finding container b8aeac9037c00c48c6c9f723efb6dfdddcb1c17cbbf81f51489367b0bd561a36: Status 404 returned error can't find the container with id b8aeac9037c00c48c6c9f723efb6dfdddcb1c17cbbf81f51489367b0bd561a36 Oct 30 01:21:12.080638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.080608 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m"] Oct 30 01:21:12.080765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.080727 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:12.080845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.080814 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:12.080901 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.080854 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:12.299555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.299507 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw"] Oct 30 01:21:12.335095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.335001 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw"] Oct 30 01:21:12.335265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.335145 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.460540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.439912 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.474705 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.474768 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0244d952-9b69-4cdd-bd71-349849e060a5-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.474820 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.474984 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0244d952-9b69-4cdd-bd71-349849e060a5-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.475139 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqz5x\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-kube-api-access-rqz5x\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.475181 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.475273 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.475843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.475331 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.493159 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:21:12.493124 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:21:12.577602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.577567 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqz5x\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-kube-api-access-rqz5x\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.577769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.577627 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.577769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.577696 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.577769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.577750 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.577956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.577865 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.577962 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.577999 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0244d952-9b69-4cdd-bd71-349849e060a5-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.578047 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.578082 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.578161 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.578267 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.578355 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0244d952-9b69-4cdd-bd71-349849e060a5-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.578409 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.578868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.578843 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0244d952-9b69-4cdd-bd71-349849e060a5-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.581063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.581040 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.581179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.581040 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0244d952-9b69-4cdd-bd71-349849e060a5-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.598649 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.598594 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqz5x\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-kube-api-access-rqz5x\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.598813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.598797 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.657117 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.657083 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:12.691918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.691876 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" event={"ID":"e9ea456d-9106-466b-b8eb-bea896ccd4b4","Type":"ContainerStarted","Data":"f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0"} Oct 30 01:21:12.692067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.691937 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" event={"ID":"e9ea456d-9106-466b-b8eb-bea896ccd4b4","Type":"ContainerStarted","Data":"b8aeac9037c00c48c6c9f723efb6dfdddcb1c17cbbf81f51489367b0bd561a36"} Oct 30 01:21:12.748633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.748580 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podStartSLOduration=1.748560468 podStartE2EDuration="1.748560468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:12.739929105 +0000 UTC m=+2557.683102297" watchObservedRunningTime="2026-10-30 01:21:12.748560468 +0000 UTC m=+2557.691733663" Oct 30 01:21:12.757643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:12.756950 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:13.005934 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:13.005901 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0244d952_9b69_4cdd_bd71_349849e060a5.slice/crio-1ca94fcc19adcb71f649e27f3287915455b780f9e6669d736deee480b27de36f WatchSource:0}: Error finding container 1ca94fcc19adcb71f649e27f3287915455b780f9e6669d736deee480b27de36f: Status 404 returned error can't find the container with id 1ca94fcc19adcb71f649e27f3287915455b780f9e6669d736deee480b27de36f Oct 30 01:21:13.007703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.007680 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw"] Oct 30 01:21:13.009948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.009907 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:13.010038 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.009998 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:13.010084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.010051 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:13.702347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.702305 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" event={"ID":"0244d952-9b69-4cdd-bd71-349849e060a5","Type":"ContainerStarted","Data":"8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22"} Oct 30 01:21:13.702853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.702373 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" event={"ID":"0244d952-9b69-4cdd-bd71-349849e060a5","Type":"ContainerStarted","Data":"1ca94fcc19adcb71f649e27f3287915455b780f9e6669d736deee480b27de36f"} Oct 30 01:21:13.784714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.784677 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:21:13.784988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:13.784958 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 30 01:21:14.663632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:14.663584 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:14.669707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:14.669665 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 30 01:21:14.669956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:14.669923 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 30 01:21:14.773545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:14.766277 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:21:14.773545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:14.766337 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 30 01:21:15.666598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:15.666203 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 30 01:21:15.666598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:15.666261 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 30 01:21:15.770541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:15.761909 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:21:15.770541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:15.761980 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 30 01:21:16.609883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.609854 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:21:16.610341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.610133 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:21:16.680545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.673641 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 30 01:21:16.680545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.673702 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 30 01:21:16.684878 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.683931 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podStartSLOduration=4.68391216 podStartE2EDuration="4.68391216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:13.780179141 +0000 UTC m=+2558.723352333" watchObservedRunningTime="2026-10-30 01:21:16.68391216 +0000 UTC m=+2561.627085351" Oct 30 01:21:16.762542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.757871 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:21:16.762542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.757928 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 30 01:21:16.821244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.817613 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6"] Oct 30 01:21:16.821244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:16.817853 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" containerID="cri-o://b9b5ae3cacba9a885898a340b60b04e020287a8a267615087d7815b42cc06422" gracePeriod=30 Oct 30 01:21:17.690547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:17.687675 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 30 01:21:17.690547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:17.687737 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 30 01:21:17.792542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:17.788711 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:21:17.792542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:17.788817 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 30 01:21:18.659544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:18.659117 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 30 01:21:18.659544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:18.659175 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 30 01:21:18.754541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:18.753653 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:21:18.754541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:18.753708 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:21:18.756454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:18.756285 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:21:18.756454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:18.756363 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 30 01:21:19.743643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:19.686791 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 30 01:21:19.743643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:19.686849 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 30 01:21:19.850988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:19.848451 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:19.850988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:19.850400 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:19.861678 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:19.861653 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:21:20.846033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:20.845990 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:20.848011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:20.847900 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:20.849273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:20.849254 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:21:21.010712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.005359 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r"] Oct 30 01:21:21.014927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.014892 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075160 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxjz\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-kube-api-access-2xxjz\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075256 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-socket\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075328 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-envoy\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075408 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d7288-6855-4519-be9e-ab84d52bbb57-istiod-ca-cert\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075537 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-podinfo\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075582 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-certs\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075658 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-credential-socket\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075746 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-token\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.077549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.075974 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-data\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.095064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.089975 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-96bd13-openshift-default-dockercfg-588rq\"" Oct 30 01:21:21.122860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.122828 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r"] Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.219602 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-podinfo\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.219663 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-certs\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.219725 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-credential-socket\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.219830 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-token\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.220024 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-data\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.220082 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxjz\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-kube-api-access-2xxjz\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.220144 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-socket\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.224548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.220204 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-envoy\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.270761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.233592 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d7288-6855-4519-be9e-ab84d52bbb57-istiod-ca-cert\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.270761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.234633 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d7288-6855-4519-be9e-ab84d52bbb57-istiod-ca-cert\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.270761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.235452 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-credential-socket\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.270761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.235703 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-certs\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.270761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.235915 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-socket\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.418606 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.418262 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-data\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.418838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.418816 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-token\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.419448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.419424 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxjz\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-kube-api-access-2xxjz\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.420016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.419996 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-podinfo\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.420231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.420213 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-envoy\") pod \"e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:21.711997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:21.709340 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:22.090654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.090059 4905 generic.go:356] "Generic (PLEG): container finished" podID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerID="b9b5ae3cacba9a885898a340b60b04e020287a8a267615087d7815b42cc06422" exitCode=0 Oct 30 01:21:22.091270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.090645 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" event={"ID":"0cc90b0e-4f0c-4104-8860-849aaf6772dc","Type":"ContainerDied","Data":"b9b5ae3cacba9a885898a340b60b04e020287a8a267615087d7815b42cc06422"} Oct 30 01:21:22.091270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.090736 4905 scope.go:122] "RemoveContainer" containerID="f090cbb7561550b044e6daf15c65c84321903c3b4086737276882711a88e8a3c" Oct 30 01:21:22.183353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.183306 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r"] Oct 30 01:21:22.204358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.204082 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:22.204358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.204176 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:22.204358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.204218 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:22.267205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.267176 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:21:22.311644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.311619 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-token\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.312129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.312110 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-podinfo\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.317771 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.317710 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-data\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.317890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.317817 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-certs\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.317890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.317863 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-envoy\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.318029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.317940 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istiod-ca-cert\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.318090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.318068 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-kube-api-access-vtq4x\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-kube-api-access-vtq4x\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.318149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.318098 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-socket\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.318240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.318219 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-credential-socket\") pod \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\" (UID: \"0cc90b0e-4f0c-4104-8860-849aaf6772dc\") " Oct 30 01:21:22.320407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.316053 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-token" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:22.320407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.318412 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-podinfo" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:22.320407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.319448 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-credential-socket" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:22.320407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.319762 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istiod-ca-cert" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:22.320407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.320019 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-socket" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:22.320407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.320299 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-envoy" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:22.320407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.320381 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-certs" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:22.321074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.321055 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-data" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:22.321859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.321831 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-kube-api-access-vtq4x" pod "0cc90b0e-4f0c-4104-8860-849aaf6772dc" (UID: "0cc90b0e-4f0c-4104-8860-849aaf6772dc"). InnerVolumeSpecName "kube-api-access-vtq4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:22.429473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429362 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtq4x\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-kube-api-access-vtq4x\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.429473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429402 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.429473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429419 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.429473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429435 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.429473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429449 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.429473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429462 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.429473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429477 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.434146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429489 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:22.434146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:22.429502 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc90b0e-4f0c-4104-8860-849aaf6772dc-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:23.104562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.102450 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" event={"ID":"0cc90b0e-4f0c-4104-8860-849aaf6772dc","Type":"ContainerDied","Data":"89bca070714d653e122dd426c607b42c4eadd59389f724efdaabf39173b6b6ad"} Oct 30 01:21:23.104562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.102505 4905 scope.go:122] "RemoveContainer" containerID="b9b5ae3cacba9a885898a340b60b04e020287a8a267615087d7815b42cc06422" Oct 30 01:21:23.104562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.102613 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6" Oct 30 01:21:23.117814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.107343 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" event={"ID":"b44d7288-6855-4519-be9e-ab84d52bbb57","Type":"ContainerStarted","Data":"a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605"} Oct 30 01:21:23.117814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.107412 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" event={"ID":"b44d7288-6855-4519-be9e-ab84d52bbb57","Type":"ContainerStarted","Data":"86cf8cbf0773db87bdce52073610682a1dd88a89982e3563074594bb631d9a06"} Oct 30 01:21:23.168447 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.168418 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6"] Oct 30 01:21:23.173274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.173246 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-twqt6"] Oct 30 01:21:23.196001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.195967 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" path="/var/lib/kubelet/pods/0cc90b0e-4f0c-4104-8860-849aaf6772dc/volumes" Oct 30 01:21:23.212565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.212491 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podStartSLOduration=3.212477469 podStartE2EDuration="3.212477469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:23.207007622 +0000 UTC m=+2568.150180816" watchObservedRunningTime="2026-10-30 01:21:23.212477469 +0000 UTC m=+2568.155650661" Oct 30 01:21:23.711087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.711057 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:23.717757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:23.717735 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:24.141613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:24.141581 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:24.159554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:24.155993 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:21:25.179328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:25.179293 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh"] Oct 30 01:21:25.216076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:25.216050 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd"] Oct 30 01:21:27.540793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:27.540749 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:21:27.541207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:27.540829 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:21:28.837539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.831060 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg"] Oct 30 01:21:28.869357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.869318 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" Oct 30 01:21:28.869357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.869352 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" Oct 30 01:21:28.869761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.869374 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" Oct 30 01:21:28.869761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.869382 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" Oct 30 01:21:28.869761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.869615 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" Oct 30 01:21:28.869761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.869630 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc90b0e-4f0c-4104-8860-849aaf6772dc" containerName="istio-proxy" Oct 30 01:21:28.877769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.875792 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:28.899188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.899158 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96bd13\"" Oct 30 01:21:28.899802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.899781 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96bd13-dockercfg-bhskd\"" Oct 30 01:21:28.944258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.944230 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg"] Oct 30 01:21:28.964151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.963608 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm"] Oct 30 01:21:28.973493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:28.973324 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj"] Oct 30 01:21:29.025173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.025134 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzx2\" (UniqueName: \"kubernetes.io/projected/db19fe18-fec2-4511-951b-d612232bda2f-kube-api-access-2fzx2\") pod \"payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:29.025311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.025285 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db19fe18-fec2-4511-951b-d612232bda2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:29.128748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.128663 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzx2\" (UniqueName: \"kubernetes.io/projected/db19fe18-fec2-4511-951b-d612232bda2f-kube-api-access-2fzx2\") pod \"payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:29.128952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.128798 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db19fe18-fec2-4511-951b-d612232bda2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:29.129760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.129735 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db19fe18-fec2-4511-951b-d612232bda2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:29.149997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.149968 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzx2\" (UniqueName: \"kubernetes.io/projected/db19fe18-fec2-4511-951b-d612232bda2f-kube-api-access-2fzx2\") pod \"payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:29.256576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.255478 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:29.908503 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:29.908460 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb19fe18_fec2_4511_951b_d612232bda2f.slice/crio-377a50ccf4fe485d0cd4f57b3ec04e2e6e0e580d8844a3096ab40fd2f87da07f WatchSource:0}: Error finding container 377a50ccf4fe485d0cd4f57b3ec04e2e6e0e580d8844a3096ab40fd2f87da07f: Status 404 returned error can't find the container with id 377a50ccf4fe485d0cd4f57b3ec04e2e6e0e580d8844a3096ab40fd2f87da07f Oct 30 01:21:29.910779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:29.910752 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg"] Oct 30 01:21:30.179983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.179892 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" podUID="037dba09-fa60-4a54-a692-bd142db35f83" containerName="payload-processing" containerID="cri-o://b9ebf3d6e2e17dd41b74ac80effd94a354418f25d89b1f7a8a8fd9d60c6d94ce" gracePeriod=25 Oct 30 01:21:30.217112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.217074 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" podUID="60f6e4bb-e4b5-4300-ad76-9ed24889bae3" containerName="payload-pre-processing" containerID="cri-o://00b2796c7126f86e3e260198b0cb0ac72f2a8787a9b11a67864677dfe234dc64" gracePeriod=25 Oct 30 01:21:30.408539 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.408490 4905 generic.go:356] "Generic (PLEG): container finished" podID="037dba09-fa60-4a54-a692-bd142db35f83" containerID="b9ebf3d6e2e17dd41b74ac80effd94a354418f25d89b1f7a8a8fd9d60c6d94ce" exitCode=0 Oct 30 01:21:30.408726 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.408596 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" event={"ID":"037dba09-fa60-4a54-a692-bd142db35f83","Type":"ContainerDied","Data":"b9ebf3d6e2e17dd41b74ac80effd94a354418f25d89b1f7a8a8fd9d60c6d94ce"} Oct 30 01:21:30.410593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.410561 4905 generic.go:356] "Generic (PLEG): container finished" podID="60f6e4bb-e4b5-4300-ad76-9ed24889bae3" containerID="00b2796c7126f86e3e260198b0cb0ac72f2a8787a9b11a67864677dfe234dc64" exitCode=0 Oct 30 01:21:30.410737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.410679 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" event={"ID":"60f6e4bb-e4b5-4300-ad76-9ed24889bae3","Type":"ContainerDied","Data":"00b2796c7126f86e3e260198b0cb0ac72f2a8787a9b11a67864677dfe234dc64"} Oct 30 01:21:30.412974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.412906 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" event={"ID":"db19fe18-fec2-4511-951b-d612232bda2f","Type":"ContainerStarted","Data":"94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1"} Oct 30 01:21:30.413258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.413237 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" event={"ID":"db19fe18-fec2-4511-951b-d612232bda2f","Type":"ContainerStarted","Data":"377a50ccf4fe485d0cd4f57b3ec04e2e6e0e580d8844a3096ab40fd2f87da07f"} Oct 30 01:21:30.413584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.413567 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:30.472299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.472241 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" podStartSLOduration=2.47222236 podStartE2EDuration="2.47222236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:30.471043528 +0000 UTC m=+2575.414216720" watchObservedRunningTime="2026-10-30 01:21:30.47222236 +0000 UTC m=+2575.415395552" Oct 30 01:21:30.693948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.693908 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:21:30.723717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.723658 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:21:30.888540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.888499 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-kube-api-access-dc9sl\" (UniqueName: \"kubernetes.io/projected/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-kube-api-access-dc9sl\") pod \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " Oct 30 01:21:30.888738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.888604 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037dba09-fa60-4a54-a692-bd142db35f83-kube-api-access-qhsth\" (UniqueName: \"kubernetes.io/projected/037dba09-fa60-4a54-a692-bd142db35f83-kube-api-access-qhsth\") pod \"037dba09-fa60-4a54-a692-bd142db35f83\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " Oct 30 01:21:30.888738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.888722 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037dba09-fa60-4a54-a692-bd142db35f83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dba09-fa60-4a54-a692-bd142db35f83-plugins-config-volume\") pod \"037dba09-fa60-4a54-a692-bd142db35f83\" (UID: \"037dba09-fa60-4a54-a692-bd142db35f83\") " Oct 30 01:21:30.888857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.888825 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-plugins-config-volume\") pod \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\" (UID: \"60f6e4bb-e4b5-4300-ad76-9ed24889bae3\") " Oct 30 01:21:30.889218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.889109 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037dba09-fa60-4a54-a692-bd142db35f83-plugins-config-volume" pod "037dba09-fa60-4a54-a692-bd142db35f83" (UID: "037dba09-fa60-4a54-a692-bd142db35f83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:30.889341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.889288 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-plugins-config-volume" pod "60f6e4bb-e4b5-4300-ad76-9ed24889bae3" (UID: "60f6e4bb-e4b5-4300-ad76-9ed24889bae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:30.889763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.889741 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dba09-fa60-4a54-a692-bd142db35f83-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:30.889846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.889768 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:30.890981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.890957 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-kube-api-access-dc9sl" pod "60f6e4bb-e4b5-4300-ad76-9ed24889bae3" (UID: "60f6e4bb-e4b5-4300-ad76-9ed24889bae3"). InnerVolumeSpecName "kube-api-access-dc9sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:30.891369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.891355 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037dba09-fa60-4a54-a692-bd142db35f83-kube-api-access-qhsth" pod "037dba09-fa60-4a54-a692-bd142db35f83" (UID: "037dba09-fa60-4a54-a692-bd142db35f83"). InnerVolumeSpecName "kube-api-access-qhsth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:30.990943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.990911 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc9sl\" (UniqueName: \"kubernetes.io/projected/60f6e4bb-e4b5-4300-ad76-9ed24889bae3-kube-api-access-dc9sl\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:30.990943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:30.990937 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhsth\" (UniqueName: \"kubernetes.io/projected/037dba09-fa60-4a54-a692-bd142db35f83-kube-api-access-qhsth\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:31.426729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.425809 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" event={"ID":"037dba09-fa60-4a54-a692-bd142db35f83","Type":"ContainerDied","Data":"993d2986606c7d2c067af394e3abe798a050321613517f280d3670af08289d4d"} Oct 30 01:21:31.426729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.425881 4905 scope.go:122] "RemoveContainer" containerID="b9ebf3d6e2e17dd41b74ac80effd94a354418f25d89b1f7a8a8fd9d60c6d94ce" Oct 30 01:21:31.426729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.426090 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh" Oct 30 01:21:31.429050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.429010 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" Oct 30 01:21:31.429250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.429224 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd" event={"ID":"60f6e4bb-e4b5-4300-ad76-9ed24889bae3","Type":"ContainerDied","Data":"7ec097788bd5b63baa7be33c678eab7694f59dd458e78c05d3d0270f78ea223b"} Oct 30 01:21:31.445056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.444995 4905 scope.go:122] "RemoveContainer" containerID="00b2796c7126f86e3e260198b0cb0ac72f2a8787a9b11a67864677dfe234dc64" Oct 30 01:21:31.487166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.487140 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh"] Oct 30 01:21:31.500089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.500066 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-7d9867d786-wd6qh"] Oct 30 01:21:31.527115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.527054 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd"] Oct 30 01:21:31.534242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:31.534217 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-d6965467f-8fjfd"] Oct 30 01:21:32.440839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:32.440802 4905 generic.go:356] "Generic (PLEG): container finished" podID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerID="179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141" exitCode=0 Oct 30 01:21:32.441311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:32.440880 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" event={"ID":"8aaf339a-6d1f-460e-8b17-57b3494d6707","Type":"ContainerDied","Data":"179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141"} Oct 30 01:21:32.441396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:32.441378 4905 scope.go:122] "RemoveContainer" containerID="179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141" Oct 30 01:21:32.444443 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:32.444410 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:32.444635 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:32.444482 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:32.444635 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:32.444531 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:33.206542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:33.203097 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037dba09-fa60-4a54-a692-bd142db35f83" path="/var/lib/kubelet/pods/037dba09-fa60-4a54-a692-bd142db35f83/volumes" Oct 30 01:21:33.206542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:33.203974 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f6e4bb-e4b5-4300-ad76-9ed24889bae3" path="/var/lib/kubelet/pods/60f6e4bb-e4b5-4300-ad76-9ed24889bae3/volumes" Oct 30 01:21:33.472624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:33.471105 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" event={"ID":"8aaf339a-6d1f-460e-8b17-57b3494d6707","Type":"ContainerStarted","Data":"00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907"} Oct 30 01:21:33.964404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:33.964345 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" podUID="db932632-d98d-4b76-9be3-318e2be88e3a" containerName="payload-processing" containerID="cri-o://69675fefb7a7421f38687c218ac74d5505c071edf1ab8724e77a3f64315f6b33" gracePeriod=25 Oct 30 01:21:33.974168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:33.974136 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" podUID="32177166-9202-4892-973a-99115094adc9" containerName="payload-pre-processing" containerID="cri-o://a935c0805bc64d424ad0303b7cdc8785d2b4df3524ce580fb2f0113e2b4cd407" gracePeriod=25 Oct 30 01:21:34.206571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:34.206514 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:21:34.207230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:34.207210 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:21:34.207283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:34.207258 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 01:21:35.207381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:35.207343 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:21:35.207824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:35.207400 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 01:21:36.207422 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:36.207393 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:21:36.207829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:36.207439 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 01:21:37.206765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:37.206732 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:21:37.206959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:37.206792 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 01:21:38.209587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:38.207861 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:21:38.209587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:38.207925 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 01:21:39.212556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:39.212509 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:21:39.213031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:39.213009 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:21:39.213639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:39.213617 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:21:39.801933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:39.801896 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" podUID="db932632-d98d-4b76-9be3-318e2be88e3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:39.802135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:39.801988 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" podUID="32177166-9202-4892-973a-99115094adc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:41.657177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.657134 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc"] Oct 30 01:21:41.660667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.660643 4905 cpu_manager.go:414] "containerMap: removing container" podUID="60f6e4bb-e4b5-4300-ad76-9ed24889bae3" containerName="payload-pre-processing" Oct 30 01:21:41.660845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.660826 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f6e4bb-e4b5-4300-ad76-9ed24889bae3" containerName="payload-pre-processing" Oct 30 01:21:41.661036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.661009 4905 cpu_manager.go:414] "containerMap: removing container" podUID="037dba09-fa60-4a54-a692-bd142db35f83" containerName="payload-processing" Oct 30 01:21:41.661163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.661148 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037dba09-fa60-4a54-a692-bd142db35f83" containerName="payload-processing" Oct 30 01:21:41.662937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.662919 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="037dba09-fa60-4a54-a692-bd142db35f83" containerName="payload-processing" Oct 30 01:21:41.663627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.663600 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f6e4bb-e4b5-4300-ad76-9ed24889bae3" containerName="payload-pre-processing" Oct 30 01:21:41.671869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.671850 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:21:41.672770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.672096 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.674346 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.674299 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc"] Oct 30 01:21:41.933274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933064 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933177 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933258 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxrm\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-kube-api-access-qtxrm\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933382 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933443 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933493 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933743 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933895 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:41.933997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:41.933933 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035381 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035410 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035458 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035485 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035567 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxrm\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-kube-api-access-qtxrm\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035598 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035619 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035660 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.035832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035825 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.036002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.035870 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.036186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.036161 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.036269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.036196 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.036480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.036462 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.036598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.036583 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.038003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.037978 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.038155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.038139 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.056227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.056207 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxrm\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-kube-api-access-qtxrm\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.056449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.056427 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.091252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.091229 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:42.414129 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:42.414074 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf94bd3_19e1_4e64_85ca_bda6f576079d.slice/crio-38d3603fb05f20b1bf13b26082acd4d7fc9c9c042b1f5ddb7fce96a95d5cb959 WatchSource:0}: Error finding container 38d3603fb05f20b1bf13b26082acd4d7fc9c9c042b1f5ddb7fce96a95d5cb959: Status 404 returned error can't find the container with id 38d3603fb05f20b1bf13b26082acd4d7fc9c9c042b1f5ddb7fce96a95d5cb959 Oct 30 01:21:42.437307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.437230 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc"] Oct 30 01:21:42.442717 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:21:42.442678 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:21:42.443853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.443813 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:42.443976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.443887 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:42.443976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.443918 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:21:42.831856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.831796 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" event={"ID":"5cf94bd3-19e1-4e64-85ca-bda6f576079d","Type":"ContainerStarted","Data":"58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a"} Oct 30 01:21:42.831856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.831853 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" event={"ID":"5cf94bd3-19e1-4e64-85ca-bda6f576079d","Type":"ContainerStarted","Data":"38d3603fb05f20b1bf13b26082acd4d7fc9c9c042b1f5ddb7fce96a95d5cb959"} Oct 30 01:21:42.890897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:42.890847 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podStartSLOduration=1.890831557 podStartE2EDuration="1.890831557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:42.889923609 +0000 UTC m=+2587.833096813" watchObservedRunningTime="2026-10-30 01:21:42.890831557 +0000 UTC m=+2587.834004751" Oct 30 01:21:43.092595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:43.092437 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:43.094681 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:43.094642 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 01:21:43.094852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:43.094701 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 01:21:44.092389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:44.092350 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 01:21:44.092906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:44.092407 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 01:21:45.091597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:45.091563 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 01:21:45.091791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:45.091619 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 01:21:46.091945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:46.091915 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 01:21:46.092357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:46.091965 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 01:21:47.093299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:47.093262 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 01:21:47.094148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:47.094078 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 01:21:48.092548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:48.092332 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 01:21:48.092548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:48.092390 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 01:21:49.096899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:49.096867 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:49.097452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:49.097432 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:49.098126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:49.098103 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:21:49.802106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:49.802066 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" podUID="32177166-9202-4892-973a-99115094adc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:49.802258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:49.802133 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" podUID="db932632-d98d-4b76-9be3-318e2be88e3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:55.282296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.282258 4905 generic.go:356] "Generic (PLEG): container finished" podID="db932632-d98d-4b76-9be3-318e2be88e3a" containerID="69675fefb7a7421f38687c218ac74d5505c071edf1ab8724e77a3f64315f6b33" exitCode=0 Oct 30 01:21:55.304138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.304107 4905 generic.go:356] "Generic (PLEG): container finished" podID="32177166-9202-4892-973a-99115094adc9" containerID="a935c0805bc64d424ad0303b7cdc8785d2b4df3524ce580fb2f0113e2b4cd407" exitCode=0 Oct 30 01:21:55.321848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.321809 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh"] Oct 30 01:21:55.364644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.363053 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" event={"ID":"db932632-d98d-4b76-9be3-318e2be88e3a","Type":"ContainerDied","Data":"69675fefb7a7421f38687c218ac74d5505c071edf1ab8724e77a3f64315f6b33"} Oct 30 01:21:55.364644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.363101 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" event={"ID":"32177166-9202-4892-973a-99115094adc9","Type":"ContainerDied","Data":"a935c0805bc64d424ad0303b7cdc8785d2b4df3524ce580fb2f0113e2b4cd407"} Oct 30 01:21:55.364644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.363131 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh"] Oct 30 01:21:55.364644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.363162 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4"] Oct 30 01:21:55.364644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.363172 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.369558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.368683 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4"] Oct 30 01:21:55.369558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.368780 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:55.391057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.391033 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:21:55.393704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.393679 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-07a3a8\"" Oct 30 01:21:55.393817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.393721 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-07a3a8-dockercfg-q2tgz\"" Oct 30 01:21:55.479337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.479308 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db932632-d98d-4b76-9be3-318e2be88e3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db932632-d98d-4b76-9be3-318e2be88e3a-plugins-config-volume\") pod \"db932632-d98d-4b76-9be3-318e2be88e3a\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " Oct 30 01:21:55.479540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.479509 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db932632-d98d-4b76-9be3-318e2be88e3a-kube-api-access-f444w\" (UniqueName: \"kubernetes.io/projected/db932632-d98d-4b76-9be3-318e2be88e3a-kube-api-access-f444w\") pod \"db932632-d98d-4b76-9be3-318e2be88e3a\" (UID: \"db932632-d98d-4b76-9be3-318e2be88e3a\") " Oct 30 01:21:55.479722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.479696 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db932632-d98d-4b76-9be3-318e2be88e3a-plugins-config-volume" pod "db932632-d98d-4b76-9be3-318e2be88e3a" (UID: "db932632-d98d-4b76-9be3-318e2be88e3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:55.479813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.479798 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bj4b\" (UniqueName: \"kubernetes.io/projected/3f36f245-3b68-4918-8109-d80b64f270a8-kube-api-access-6bj4b\") pod \"payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:55.479889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.479855 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pv5\" (UniqueName: \"kubernetes.io/projected/5b0c33c3-8280-406c-ae34-b032c03b8432-kube-api-access-49pv5\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.480073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.480051 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c33c3-8280-406c-ae34-b032c03b8432-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.480365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.480338 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36f245-3b68-4918-8109-d80b64f270a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:55.480580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.480561 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db932632-d98d-4b76-9be3-318e2be88e3a-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:55.481812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.481789 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db932632-d98d-4b76-9be3-318e2be88e3a-kube-api-access-f444w" pod "db932632-d98d-4b76-9be3-318e2be88e3a" (UID: "db932632-d98d-4b76-9be3-318e2be88e3a"). InnerVolumeSpecName "kube-api-access-f444w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:55.516243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.516221 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:21:55.582023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.581950 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32177166-9202-4892-973a-99115094adc9-kube-api-access-j7nd9\" (UniqueName: \"kubernetes.io/projected/32177166-9202-4892-973a-99115094adc9-kube-api-access-j7nd9\") pod \"32177166-9202-4892-973a-99115094adc9\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " Oct 30 01:21:55.582400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.582377 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32177166-9202-4892-973a-99115094adc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32177166-9202-4892-973a-99115094adc9-plugins-config-volume\") pod \"32177166-9202-4892-973a-99115094adc9\" (UID: \"32177166-9202-4892-973a-99115094adc9\") " Oct 30 01:21:55.582691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.582673 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pv5\" (UniqueName: \"kubernetes.io/projected/5b0c33c3-8280-406c-ae34-b032c03b8432-kube-api-access-49pv5\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.582857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.582840 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c33c3-8280-406c-ae34-b032c03b8432-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.583008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.582976 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32177166-9202-4892-973a-99115094adc9-plugins-config-volume" pod "32177166-9202-4892-973a-99115094adc9" (UID: "32177166-9202-4892-973a-99115094adc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:55.583162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.583136 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36f245-3b68-4918-8109-d80b64f270a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:55.583295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.583280 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bj4b\" (UniqueName: \"kubernetes.io/projected/3f36f245-3b68-4918-8109-d80b64f270a8-kube-api-access-6bj4b\") pod \"payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:55.583428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.583410 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f444w\" (UniqueName: \"kubernetes.io/projected/db932632-d98d-4b76-9be3-318e2be88e3a-kube-api-access-f444w\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:55.583494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.583433 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32177166-9202-4892-973a-99115094adc9-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:55.583938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.583913 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c33c3-8280-406c-ae34-b032c03b8432-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.584032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.583985 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36f245-3b68-4918-8109-d80b64f270a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:55.584732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.584696 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32177166-9202-4892-973a-99115094adc9-kube-api-access-j7nd9" pod "32177166-9202-4892-973a-99115094adc9" (UID: "32177166-9202-4892-973a-99115094adc9"). InnerVolumeSpecName "kube-api-access-j7nd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:55.603926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.603901 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pv5\" (UniqueName: \"kubernetes.io/projected/5b0c33c3-8280-406c-ae34-b032c03b8432-kube-api-access-49pv5\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.604042 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bj4b\" (UniqueName: \"kubernetes.io/projected/3f36f245-3b68-4918-8109-d80b64f270a8-kube-api-access-6bj4b\") pod \"payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:55.684935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.684907 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7nd9\" (UniqueName: \"kubernetes.io/projected/32177166-9202-4892-973a-99115094adc9-kube-api-access-j7nd9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:21:55.710105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.710079 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:55.718220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:55.718199 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:56.111277 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:56.111242 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0c33c3_8280_406c_ae34_b032c03b8432.slice/crio-966c013b4bbc03051e3b79e4eb8087f77d05eec8dd8a38512e39522c5f909bb2 WatchSource:0}: Error finding container 966c013b4bbc03051e3b79e4eb8087f77d05eec8dd8a38512e39522c5f909bb2: Status 404 returned error can't find the container with id 966c013b4bbc03051e3b79e4eb8087f77d05eec8dd8a38512e39522c5f909bb2 Oct 30 01:21:56.113434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.113409 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh"] Oct 30 01:21:56.129466 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:21:56.129442 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f36f245_3b68_4918_8109_d80b64f270a8.slice/crio-b23a32bfdde482165467b14d00ecdd1c2ca32ee65a1e67d1d9ca8f579ab76e3e WatchSource:0}: Error finding container b23a32bfdde482165467b14d00ecdd1c2ca32ee65a1e67d1d9ca8f579ab76e3e: Status 404 returned error can't find the container with id b23a32bfdde482165467b14d00ecdd1c2ca32ee65a1e67d1d9ca8f579ab76e3e Oct 30 01:21:56.131876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.131848 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4"] Oct 30 01:21:56.316040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.316014 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" Oct 30 01:21:56.316656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.316623 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm" event={"ID":"db932632-d98d-4b76-9be3-318e2be88e3a","Type":"ContainerDied","Data":"5916a3296d5388bdd42bbd2783dd94395708727007d5d3d7fd741aff043e43d7"} Oct 30 01:21:56.316768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.316700 4905 scope.go:122] "RemoveContainer" containerID="69675fefb7a7421f38687c218ac74d5505c071edf1ab8724e77a3f64315f6b33" Oct 30 01:21:56.319370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.319331 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" event={"ID":"3f36f245-3b68-4918-8109-d80b64f270a8","Type":"ContainerStarted","Data":"ef0d62b5a365b93a2429d62efc81813c13b902147d16e359e6df69c476491af9"} Oct 30 01:21:56.319508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.319386 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" event={"ID":"3f36f245-3b68-4918-8109-d80b64f270a8","Type":"ContainerStarted","Data":"b23a32bfdde482165467b14d00ecdd1c2ca32ee65a1e67d1d9ca8f579ab76e3e"} Oct 30 01:21:56.319757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.319696 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:21:56.324601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.321272 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" Oct 30 01:21:56.324601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.321292 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj" event={"ID":"32177166-9202-4892-973a-99115094adc9","Type":"ContainerDied","Data":"3a8275c0f0dcb229bdcad71f931fc7dd3b5a88ea235fe531e1f666d90198a4de"} Oct 30 01:21:56.324601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.323641 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" event={"ID":"5b0c33c3-8280-406c-ae34-b032c03b8432","Type":"ContainerStarted","Data":"6705f580716c0a6ac0f35db7bc59873f91aa7cbcc3fc21931c85e4d19c9834f3"} Oct 30 01:21:56.324601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.323688 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:21:56.324601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.323715 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" event={"ID":"5b0c33c3-8280-406c-ae34-b032c03b8432","Type":"ContainerStarted","Data":"966c013b4bbc03051e3b79e4eb8087f77d05eec8dd8a38512e39522c5f909bb2"} Oct 30 01:21:56.368707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.368465 4905 scope.go:122] "RemoveContainer" containerID="a935c0805bc64d424ad0303b7cdc8785d2b4df3524ce580fb2f0113e2b4cd407" Oct 30 01:21:56.382559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.382499 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" podStartSLOduration=1.3824863139999999 podStartE2EDuration="1.382486314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:56.376854743 +0000 UTC m=+2601.320027938" watchObservedRunningTime="2026-10-30 01:21:56.382486314 +0000 UTC m=+2601.325659558" Oct 30 01:21:56.420080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.420038 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" podStartSLOduration=2.42002569 podStartE2EDuration="2.42002569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:56.416851707 +0000 UTC m=+2601.360024911" watchObservedRunningTime="2026-10-30 01:21:56.42002569 +0000 UTC m=+2601.363198882" Oct 30 01:21:56.487585 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.479580 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj"] Oct 30 01:21:56.487585 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.487050 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43d152-d57b44ff-x9fjj"] Oct 30 01:21:56.516786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.516761 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm"] Oct 30 01:21:56.521426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:56.521402 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43d152-6d59f45b56-hxvnm"] Oct 30 01:21:57.195922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:57.195887 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32177166-9202-4892-973a-99115094adc9" path="/var/lib/kubelet/pods/32177166-9202-4892-973a-99115094adc9/volumes" Oct 30 01:21:57.196481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:21:57.196465 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db932632-d98d-4b76-9be3-318e2be88e3a" path="/var/lib/kubelet/pods/db932632-d98d-4b76-9be3-318e2be88e3a/volumes" Oct 30 01:22:07.341345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:07.341315 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:22:07.342660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:07.342633 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:22:10.862619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:10.862576 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz"] Oct 30 01:22:10.882545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:10.864612 4905 cpu_manager.go:414] "containerMap: removing container" podUID="db932632-d98d-4b76-9be3-318e2be88e3a" containerName="payload-processing" Oct 30 01:22:10.882803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:10.882777 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db932632-d98d-4b76-9be3-318e2be88e3a" containerName="payload-processing" Oct 30 01:22:10.910543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:10.882920 4905 cpu_manager.go:414] "containerMap: removing container" podUID="32177166-9202-4892-973a-99115094adc9" containerName="payload-pre-processing" Oct 30 01:22:10.980375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:10.980337 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32177166-9202-4892-973a-99115094adc9" containerName="payload-pre-processing" Oct 30 01:22:10.990545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:10.980940 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="32177166-9202-4892-973a-99115094adc9" containerName="payload-pre-processing" Oct 30 01:22:11.048776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.048741 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="db932632-d98d-4b76-9be3-318e2be88e3a" containerName="payload-processing" Oct 30 01:22:11.091180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.091153 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.108594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.108396 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz"] Oct 30 01:22:11.142878 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.118876 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c5a66-dockercfg-bwwwk\"" Oct 30 01:22:11.143063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.132735 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c5a66\"" Oct 30 01:22:11.146412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.146382 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s"] Oct 30 01:22:11.169406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.169380 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s"] Oct 30 01:22:11.169777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.169759 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.177126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.177096 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rcz\" (UniqueName: \"kubernetes.io/projected/44210db3-5bda-495e-bdc6-70bc14176599-kube-api-access-s6rcz\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.177355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.177337 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44210db3-5bda-495e-bdc6-70bc14176599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.279348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.279313 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628e0a8b-2a09-4a0e-a8b1-580b19d88237-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.279605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.279581 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rcz\" (UniqueName: \"kubernetes.io/projected/44210db3-5bda-495e-bdc6-70bc14176599-kube-api-access-s6rcz\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.279714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.279700 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgn78\" (UniqueName: \"kubernetes.io/projected/628e0a8b-2a09-4a0e-a8b1-580b19d88237-kube-api-access-jgn78\") pod \"payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.279789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.279762 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44210db3-5bda-495e-bdc6-70bc14176599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.280491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.280472 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44210db3-5bda-495e-bdc6-70bc14176599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.300387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.300361 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rcz\" (UniqueName: \"kubernetes.io/projected/44210db3-5bda-495e-bdc6-70bc14176599-kube-api-access-s6rcz\") pod \"payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.381009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.380936 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgn78\" (UniqueName: \"kubernetes.io/projected/628e0a8b-2a09-4a0e-a8b1-580b19d88237-kube-api-access-jgn78\") pod \"payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.381146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.381027 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628e0a8b-2a09-4a0e-a8b1-580b19d88237-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.381718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.381701 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628e0a8b-2a09-4a0e-a8b1-580b19d88237-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.401351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.401322 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgn78\" (UniqueName: \"kubernetes.io/projected/628e0a8b-2a09-4a0e-a8b1-580b19d88237-kube-api-access-jgn78\") pod \"payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.430812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.430786 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:11.493437 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.493234 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:11.758281 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:22:11.758249 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44210db3_5bda_495e_bdc6_70bc14176599.slice/crio-09043268ed39d1360d10ad6304495e781dd4755075dabf16a763bbc7fe15e94c WatchSource:0}: Error finding container 09043268ed39d1360d10ad6304495e781dd4755075dabf16a763bbc7fe15e94c: Status 404 returned error can't find the container with id 09043268ed39d1360d10ad6304495e781dd4755075dabf16a763bbc7fe15e94c Oct 30 01:22:11.759688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.759661 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz"] Oct 30 01:22:11.950395 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:22:11.950344 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628e0a8b_2a09_4a0e_a8b1_580b19d88237.slice/crio-94971dba57d5cff31717d784cb650ddf1489bdc30126be36c7d5a56dbbc8b335 WatchSource:0}: Error finding container 94971dba57d5cff31717d784cb650ddf1489bdc30126be36c7d5a56dbbc8b335: Status 404 returned error can't find the container with id 94971dba57d5cff31717d784cb650ddf1489bdc30126be36c7d5a56dbbc8b335 Oct 30 01:22:11.962831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:11.953510 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s"] Oct 30 01:22:12.419285 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:12.419245 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:22:12.519925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.519885 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" event={"ID":"44210db3-5bda-495e-bdc6-70bc14176599","Type":"ContainerStarted","Data":"1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58"} Oct 30 01:22:12.520128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.519945 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" event={"ID":"44210db3-5bda-495e-bdc6-70bc14176599","Type":"ContainerStarted","Data":"09043268ed39d1360d10ad6304495e781dd4755075dabf16a763bbc7fe15e94c"} Oct 30 01:22:12.520128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.519987 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:12.521944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.521914 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" event={"ID":"628e0a8b-2a09-4a0e-a8b1-580b19d88237","Type":"ContainerStarted","Data":"633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a"} Oct 30 01:22:12.522062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.521962 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" event={"ID":"628e0a8b-2a09-4a0e-a8b1-580b19d88237","Type":"ContainerStarted","Data":"94971dba57d5cff31717d784cb650ddf1489bdc30126be36c7d5a56dbbc8b335"} Oct 30 01:22:12.522062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.522001 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:12.565282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.565239 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" podStartSLOduration=2.565225004 podStartE2EDuration="2.565225004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:12.560918645 +0000 UTC m=+2617.504091838" watchObservedRunningTime="2026-10-30 01:22:12.565225004 +0000 UTC m=+2617.508398196" Oct 30 01:22:12.606451 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:12.606409 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" podStartSLOduration=2.606395591 podStartE2EDuration="2.606395591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:12.602494796 +0000 UTC m=+2617.545667982" watchObservedRunningTime="2026-10-30 01:22:12.606395591 +0000 UTC m=+2617.549568783" Oct 30 01:22:22.210544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:22.210035 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:22:22.224550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:22.222645 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:22:22.307547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:22.307072 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:22:23.533026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:23.532995 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:22:23.533493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:23.533400 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:22:24.026133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:24.026089 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx"] Oct 30 01:22:24.026395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:24.026330 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" containerID="cri-o://0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb" gracePeriod=30 Oct 30 01:22:28.301568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:28.301498 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:22:28.302063 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:28.301608 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:22:29.376327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.376295 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:22:29.473985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.473950 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt"] Oct 30 01:22:29.487821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.487784 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9"] Oct 30 01:22:29.511818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.511788 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-token\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-token\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512012 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-socket\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512086 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-kube-api-access-sjbqz\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-kube-api-access-sjbqz\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512122 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61f6648-27e9-448f-8e07-e2ed4341a36a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61f6648-27e9-448f-8e07-e2ed4341a36a-istiod-ca-cert\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512159 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-certs\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512222 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-podinfo\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512297 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-data\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512315 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-socket" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.512353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512332 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-envoy\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512403 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-credential-socket\") pod \"e61f6648-27e9-448f-8e07-e2ed4341a36a\" (UID: \"e61f6648-27e9-448f-8e07-e2ed4341a36a\") " Oct 30 01:22:29.512756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512730 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-certs" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.512954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.512935 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-credential-socket" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.513379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.513358 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-data" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.513510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.513391 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.513738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.513716 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61f6648-27e9-448f-8e07-e2ed4341a36a-istiod-ca-cert" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:29.513888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.513872 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.514324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.514305 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.516043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.515209 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-token" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:29.516413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.516384 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-podinfo" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:29.517645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.517610 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-envoy" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.517751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.517731 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-kube-api-access-sjbqz" pod "e61f6648-27e9-448f-8e07-e2ed4341a36a" (UID: "e61f6648-27e9-448f-8e07-e2ed4341a36a"). InnerVolumeSpecName "kube-api-access-sjbqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:29.616347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.616268 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.616347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.616299 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjbqz\" (UniqueName: \"kubernetes.io/projected/e61f6648-27e9-448f-8e07-e2ed4341a36a-kube-api-access-sjbqz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.616347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.616309 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61f6648-27e9-448f-8e07-e2ed4341a36a-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.616347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.616318 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.616347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.616329 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.616347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.616339 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61f6648-27e9-448f-8e07-e2ed4341a36a-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.908367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.908286 4905 generic.go:356] "Generic (PLEG): container finished" podID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerID="0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb" exitCode=0 Oct 30 01:22:29.908367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.908339 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" Oct 30 01:22:29.908688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.908374 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" event={"ID":"e61f6648-27e9-448f-8e07-e2ed4341a36a","Type":"ContainerDied","Data":"0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb"} Oct 30 01:22:29.908688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.908441 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx" event={"ID":"e61f6648-27e9-448f-8e07-e2ed4341a36a","Type":"ContainerDied","Data":"7d1254732cc51cfeed83194203471743fd3f88f91fcd1a7c7aa0b03cc1348ee4"} Oct 30 01:22:29.908688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.908473 4905 scope.go:122] "RemoveContainer" containerID="0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb" Oct 30 01:22:29.922731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.922710 4905 scope.go:122] "RemoveContainer" containerID="9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9" Oct 30 01:22:29.936702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.936682 4905 scope.go:122] "RemoveContainer" containerID="0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb" Oct 30 01:22:29.937027 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:29.936998 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb\": container with ID starting with 0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb not found: ID does not exist" containerID="0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb" Oct 30 01:22:29.937135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.937031 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb"} err="failed to get container status \"0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb\": rpc error: code = NotFound desc = could not find container \"0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb\": container with ID starting with 0c50a6414672b410effe09e9b0548169f7e3fc8392cc5f842cc4497fbbe71ddb not found: ID does not exist" Oct 30 01:22:29.937135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.937054 4905 scope.go:122] "RemoveContainer" containerID="9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9" Oct 30 01:22:29.937301 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:29.937283 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9\": container with ID starting with 9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9 not found: ID does not exist" containerID="9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9" Oct 30 01:22:29.937364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.937306 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9"} err="failed to get container status \"9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9\": rpc error: code = NotFound desc = could not find container \"9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9\": container with ID starting with 9daa782af23005194de0f8cc969e8169649bbc959ae857f2d332fb5dd97758c9 not found: ID does not exist" Oct 30 01:22:29.969004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.968977 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx"] Oct 30 01:22:29.982923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:29.982899 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-m9ldx"] Oct 30 01:22:31.196504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:31.196470 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" path="/var/lib/kubelet/pods/e61f6648-27e9-448f-8e07-e2ed4341a36a/volumes" Oct 30 01:22:35.742461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:35.742426 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x"] Oct 30 01:22:35.742993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:35.742681 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" containerID="cri-o://7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93" gracePeriod=30 Oct 30 01:22:41.081652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.081624 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:22:41.083756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.083731 4905 generic.go:356] "Generic (PLEG): container finished" podID="8e37aa37-d267-4038-93ff-6510ffd77977" containerID="7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93" exitCode=0 Oct 30 01:22:41.083926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.083774 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" Oct 30 01:22:41.083926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.083812 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" event={"ID":"8e37aa37-d267-4038-93ff-6510ffd77977","Type":"ContainerDied","Data":"7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93"} Oct 30 01:22:41.083926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.083879 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x" event={"ID":"8e37aa37-d267-4038-93ff-6510ffd77977","Type":"ContainerDied","Data":"b81e9b81dc5b42015ce10ed15024f347635a43f61d213da4abd547b16c89fcca"} Oct 30 01:22:41.083926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.083895 4905 scope.go:122] "RemoveContainer" containerID="7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93" Oct 30 01:22:41.096342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.096324 4905 scope.go:122] "RemoveContainer" containerID="7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93" Oct 30 01:22:41.096650 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:41.096630 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93\": container with ID starting with 7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93 not found: ID does not exist" containerID="7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93" Oct 30 01:22:41.096757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.096661 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93"} err="failed to get container status \"7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93\": rpc error: code = NotFound desc = could not find container \"7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93\": container with ID starting with 7ae1b2f47e506c5728e28ebb7c829314c421a2c024d81d3c1fd332ca4cf20e93 not found: ID does not exist" Oct 30 01:22:41.175641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.175602 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-certs\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.175829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.175706 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-kube-api-access-6zb8c\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-kube-api-access-6zb8c\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.175829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.175750 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e37aa37-d267-4038-93ff-6510ffd77977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e37aa37-d267-4038-93ff-6510ffd77977-istiod-ca-cert\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.175829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.175788 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-istio-token\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-istio-token\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.176027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.175895 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e37aa37-d267-4038-93ff-6510ffd77977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e37aa37-d267-4038-93ff-6510ffd77977-istio-podinfo\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.176027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.175949 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-certs" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:41.176027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.175971 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-credential-socket\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.176206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.176167 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-envoy\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.176275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.176243 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-socket\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.176369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.176349 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-data\") pod \"8e37aa37-d267-4038-93ff-6510ffd77977\" (UID: \"8e37aa37-d267-4038-93ff-6510ffd77977\") " Oct 30 01:22:41.176724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.176693 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-credential-socket" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:41.177425 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.177398 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.177606 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.177431 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.178164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.178133 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-socket" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:41.178269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.178236 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e37aa37-d267-4038-93ff-6510ffd77977-istiod-ca-cert" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:41.178382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.178356 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-data" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:41.194840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.193056 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-istio-token" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:41.194840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.194112 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-envoy" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:41.194840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.194161 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-kube-api-access-6zb8c" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "kube-api-access-6zb8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:41.194840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.194186 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e37aa37-d267-4038-93ff-6510ffd77977-istio-podinfo" pod "8e37aa37-d267-4038-93ff-6510ffd77977" (UID: "8e37aa37-d267-4038-93ff-6510ffd77977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:41.279388 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.279358 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.279581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.279407 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.279581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.279433 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e37aa37-d267-4038-93ff-6510ffd77977-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.279581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.279451 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zb8c\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-kube-api-access-6zb8c\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.279581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.279466 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e37aa37-d267-4038-93ff-6510ffd77977-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.279581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.279480 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e37aa37-d267-4038-93ff-6510ffd77977-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.279581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.279491 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e37aa37-d267-4038-93ff-6510ffd77977-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.429914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.429811 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x"] Oct 30 01:22:41.437498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:41.437469 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-94q7x"] Oct 30 01:22:42.417457 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:42.417424 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:22:43.194891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:43.194859 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" path="/var/lib/kubelet/pods/8e37aa37-d267-4038-93ff-6510ffd77977/volumes" Oct 30 01:22:44.504971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.504926 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" containerID="cri-o://7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3" gracePeriod=45 Oct 30 01:22:44.515974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.515913 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" containerID="cri-o://36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c" gracePeriod=45 Oct 30 01:22:44.874442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.874400 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:22:44.881911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.880197 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:22:44.964186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.964150 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-home\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-home\") pod \"c6f19f52-412e-4c09-8323-6fed84a9a68b\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " Oct 30 01:22:44.969975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.969956 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d62e24-c720-4e25-a170-a5740a38ee8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/09d62e24-c720-4e25-a170-a5740a38ee8f-tls-certs\") pod \"09d62e24-c720-4e25-a170-a5740a38ee8f\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " Oct 30 01:22:44.970566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.970495 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d62e24-c720-4e25-a170-a5740a38ee8f-kube-api-access-d6z7c\" (UniqueName: \"kubernetes.io/projected/09d62e24-c720-4e25-a170-a5740a38ee8f-kube-api-access-d6z7c\") pod \"09d62e24-c720-4e25-a170-a5740a38ee8f\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " Oct 30 01:22:44.970702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.970566 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-tmp-dir\") pod \"09d62e24-c720-4e25-a170-a5740a38ee8f\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " Oct 30 01:22:44.970702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.969828 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-home" pod "c6f19f52-412e-4c09-8323-6fed84a9a68b" (UID: "c6f19f52-412e-4c09-8323-6fed84a9a68b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.970945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.970922 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-tmp-dir" pod "09d62e24-c720-4e25-a170-a5740a38ee8f" (UID: "09d62e24-c720-4e25-a170-a5740a38ee8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.971085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.971066 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f19f52-412e-4c09-8323-6fed84a9a68b-kube-api-access-fq72g\" (UniqueName: \"kubernetes.io/projected/c6f19f52-412e-4c09-8323-6fed84a9a68b-kube-api-access-fq72g\") pod \"c6f19f52-412e-4c09-8323-6fed84a9a68b\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " Oct 30 01:22:44.971194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.971123 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f19f52-412e-4c09-8323-6fed84a9a68b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f19f52-412e-4c09-8323-6fed84a9a68b-tls-certs\") pod \"c6f19f52-412e-4c09-8323-6fed84a9a68b\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " Oct 30 01:22:44.972108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.972063 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-home" pod "09d62e24-c720-4e25-a170-a5740a38ee8f" (UID: "09d62e24-c720-4e25-a170-a5740a38ee8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.972491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.972469 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-home\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-home\") pod \"09d62e24-c720-4e25-a170-a5740a38ee8f\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " Oct 30 01:22:44.972780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.972763 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-model-cache\") pod \"09d62e24-c720-4e25-a170-a5740a38ee8f\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " Oct 30 01:22:44.973169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973151 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-model-cache\") pod \"c6f19f52-412e-4c09-8323-6fed84a9a68b\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " Oct 30 01:22:44.973715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973697 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-dshm\") pod \"c6f19f52-412e-4c09-8323-6fed84a9a68b\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " Oct 30 01:22:44.973826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973737 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-dshm\") pod \"09d62e24-c720-4e25-a170-a5740a38ee8f\" (UID: \"09d62e24-c720-4e25-a170-a5740a38ee8f\") " Oct 30 01:22:44.973907 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973082 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-model-cache" pod "09d62e24-c720-4e25-a170-a5740a38ee8f" (UID: "09d62e24-c720-4e25-a170-a5740a38ee8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.973982 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973455 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-model-cache" pod "c6f19f52-412e-4c09-8323-6fed84a9a68b" (UID: "c6f19f52-412e-4c09-8323-6fed84a9a68b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.974064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973637 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d62e24-c720-4e25-a170-a5740a38ee8f-kube-api-access-d6z7c" pod "09d62e24-c720-4e25-a170-a5740a38ee8f" (UID: "09d62e24-c720-4e25-a170-a5740a38ee8f"). InnerVolumeSpecName "kube-api-access-d6z7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:44.974186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973646 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f19f52-412e-4c09-8323-6fed84a9a68b-kube-api-access-fq72g" pod "c6f19f52-412e-4c09-8323-6fed84a9a68b" (UID: "c6f19f52-412e-4c09-8323-6fed84a9a68b"). InnerVolumeSpecName "kube-api-access-fq72g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:44.974429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.974399 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f19f52-412e-4c09-8323-6fed84a9a68b-tls-certs" pod "c6f19f52-412e-4c09-8323-6fed84a9a68b" (UID: "c6f19f52-412e-4c09-8323-6fed84a9a68b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:44.974570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.973945 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-tmp-dir\") pod \"c6f19f52-412e-4c09-8323-6fed84a9a68b\" (UID: \"c6f19f52-412e-4c09-8323-6fed84a9a68b\") " Oct 30 01:22:44.975349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975329 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f19f52-412e-4c09-8323-6fed84a9a68b-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975463 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975483 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975494 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975504 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975578 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6z7c\" (UniqueName: \"kubernetes.io/projected/09d62e24-c720-4e25-a170-a5740a38ee8f-kube-api-access-d6z7c\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975594 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975609 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq72g\" (UniqueName: \"kubernetes.io/projected/c6f19f52-412e-4c09-8323-6fed84a9a68b-kube-api-access-fq72g\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:44.975788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.975660 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-tmp-dir" pod "c6f19f52-412e-4c09-8323-6fed84a9a68b" (UID: "c6f19f52-412e-4c09-8323-6fed84a9a68b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.976392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.976367 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-dshm" pod "c6f19f52-412e-4c09-8323-6fed84a9a68b" (UID: "c6f19f52-412e-4c09-8323-6fed84a9a68b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.976802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.976778 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-dshm" pod "09d62e24-c720-4e25-a170-a5740a38ee8f" (UID: "09d62e24-c720-4e25-a170-a5740a38ee8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:44.977814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:44.977793 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d62e24-c720-4e25-a170-a5740a38ee8f-tls-certs" pod "09d62e24-c720-4e25-a170-a5740a38ee8f" (UID: "09d62e24-c720-4e25-a170-a5740a38ee8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:45.077738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.077641 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:45.077738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.077665 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d62e24-c720-4e25-a170-a5740a38ee8f-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:45.077738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.077674 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f19f52-412e-4c09-8323-6fed84a9a68b-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:45.077738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.077684 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d62e24-c720-4e25-a170-a5740a38ee8f-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:22:45.274542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.267833 4905 generic.go:356] "Generic (PLEG): container finished" podID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerID="36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c" exitCode=0 Oct 30 01:22:45.296543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.290679 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" Oct 30 01:22:45.325647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.325606 4905 generic.go:356] "Generic (PLEG): container finished" podID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerID="7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3" exitCode=0 Oct 30 01:22:45.338592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.336116 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" Oct 30 01:22:45.358637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.358599 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" event={"ID":"09d62e24-c720-4e25-a170-a5740a38ee8f","Type":"ContainerDied","Data":"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c"} Oct 30 01:22:45.358897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.358875 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9" event={"ID":"09d62e24-c720-4e25-a170-a5740a38ee8f","Type":"ContainerDied","Data":"a76a33c544541ef12546d7dab1b0cfa071f567c7b16f69b66c15c536112f9362"} Oct 30 01:22:45.359018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.359003 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" event={"ID":"c6f19f52-412e-4c09-8323-6fed84a9a68b","Type":"ContainerDied","Data":"7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3"} Oct 30 01:22:45.359118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.359105 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt" event={"ID":"c6f19f52-412e-4c09-8323-6fed84a9a68b","Type":"ContainerDied","Data":"d207c2fee09d60039d1f10288509991c0936233c09eafbbfa225d189b943a849"} Oct 30 01:22:45.359204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.359194 4905 scope.go:122] "RemoveContainer" containerID="36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c" Oct 30 01:22:45.428548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.415561 4905 scope.go:122] "RemoveContainer" containerID="36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c" Oct 30 01:22:45.633634 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:45.626166 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c\": container with ID starting with 36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c not found: ID does not exist" containerID="36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c" Oct 30 01:22:45.633634 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:45.626208 4905 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c\": rpc error: code = NotFound desc = could not find container \"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c\": container with ID starting with 36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c not found: ID does not exist" containerID="36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c" Oct 30 01:22:45.633634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.626244 4905 scope.go:122] "RemoveContainer" containerID="7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3" Oct 30 01:22:45.633634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.626387 4905 scope.go:122] "RemoveContainer" containerID="36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c" Oct 30 01:22:45.758256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.747003 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c"} err="failed to get container status \"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c\": rpc error: code = NotFound desc = could not find container \"36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c\": container with ID starting with 36ac988c052899d0296f4f03257b11841f6c25b15e7fa1a40bd5c702df23ba9c not found: ID does not exist" Oct 30 01:22:45.758256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.747059 4905 scope.go:122] "RemoveContainer" containerID="7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3" Oct 30 01:22:45.835633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.835585 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt"] Oct 30 01:22:45.892039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.892004 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt"] Oct 30 01:22:45.892327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.892308 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9"] Oct 30 01:22:45.900962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.900790 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/premium-w0-3c2596-kserve-b8ddcb66b-xf4p9"] Oct 30 01:22:45.934575 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:45.931207 4905 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_main_distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt_e2e-models-e2e-worker-w0-3c2596_c6f19f52-412e-4c09-8323-6fed84a9a68b_0 in pod sandbox d207c2fee09d60039d1f10288509991c0936233c09eafbbfa225d189b943a849 from index: no such id: '7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3'" containerID="7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3" Oct 30 01:22:45.934575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.931266 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3"} err="rpc error: code = Unknown desc = failed to delete container k8s_main_distinct-2-w0-3c2596-kserve-56cb4449fd-tshnt_e2e-models-e2e-worker-w0-3c2596_c6f19f52-412e-4c09-8323-6fed84a9a68b_0 in pod sandbox d207c2fee09d60039d1f10288509991c0936233c09eafbbfa225d189b943a849 from index: no such id: '7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3'" Oct 30 01:22:45.934575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.931295 4905 scope.go:122] "RemoveContainer" containerID="7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3" Oct 30 01:22:45.934575 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:22:45.931720 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3\": container with ID starting with 7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3 not found: ID does not exist" containerID="7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3" Oct 30 01:22:45.934575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:45.931755 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3"} err="failed to get container status \"7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3\": rpc error: code = NotFound desc = could not find container \"7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3\": container with ID starting with 7517e3d4479283324abcffe6150456c590c8fd9f8db78c734bf805039f329dc3 not found: ID does not exist" Oct 30 01:22:47.195120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:47.195084 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" path="/var/lib/kubelet/pods/09d62e24-c720-4e25-a170-a5740a38ee8f/volumes" Oct 30 01:22:47.195757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:47.195738 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" path="/var/lib/kubelet/pods/c6f19f52-412e-4c09-8323-6fed84a9a68b/volumes" Oct 30 01:22:55.196672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:55.196633 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:22:55.197068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:55.196695 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:22:58.622387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:58.622302 4905 generic.go:356] "Generic (PLEG): container finished" podID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerID="a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605" exitCode=0 Oct 30 01:22:58.622905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:58.622379 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" event={"ID":"b44d7288-6855-4519-be9e-ab84d52bbb57","Type":"ContainerDied","Data":"a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605"} Oct 30 01:22:58.622905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:58.622837 4905 scope.go:122] "RemoveContainer" containerID="a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605" Oct 30 01:22:58.624840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:58.624776 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:22:58.624954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:58.624850 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:22:58.624954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:58.624882 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:22:59.638390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:59.638333 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" event={"ID":"b44d7288-6855-4519-be9e-ab84d52bbb57","Type":"ContainerStarted","Data":"de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2"} Oct 30 01:22:59.710814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:59.710782 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:22:59.710985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:59.710937 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 01:22:59.710985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:22:59.710977 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 01:23:00.710934 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:00.710897 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 01:23:00.711340 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:00.710972 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 01:23:01.710572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:01.710538 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 01:23:01.710753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:01.710591 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 01:23:02.712138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:02.712094 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 01:23:02.712599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:02.712158 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 01:23:03.571831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.571803 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:03.572036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.571855 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:03.692609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.692508 4905 generic.go:356] "Generic (PLEG): container finished" podID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerID="9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c" exitCode=0 Oct 30 01:23:03.692609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.692588 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" event={"ID":"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d","Type":"ContainerDied","Data":"9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c"} Oct 30 01:23:03.693084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.693071 4905 scope.go:122] "RemoveContainer" containerID="9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c" Oct 30 01:23:03.695006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.694976 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:03.695099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.695049 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:03.695099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.695083 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:03.710363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.710341 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 01:23:03.710484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:03.710391 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 01:23:04.702876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:04.702836 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" event={"ID":"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d","Type":"ContainerStarted","Data":"46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b"} Oct 30 01:23:04.710980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:04.710947 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 01:23:04.711107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:04.711004 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 01:23:05.106987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:05.103874 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:23:05.106987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:05.104005 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:05.106987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:05.104049 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:05.715764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:05.715736 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:23:05.716239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:05.716025 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:23:05.716928 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:05.716907 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:23:06.164701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:06.164663 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:06.164990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:06.164959 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:07.155036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:07.147764 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:07.157915 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:07.157873 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:07.345657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:07.345614 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:23:07.350808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:07.350768 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:23:08.102053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:08.102020 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:08.102280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:08.102070 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:09.101941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:09.101909 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:09.102336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:09.101961 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:10.101623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:10.101590 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:10.101835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:10.101658 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:11.102359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:11.102321 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 01:23:11.102847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:11.102395 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 01:23:12.121652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:12.120709 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:23:12.123611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:12.122444 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:23:12.124744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:12.124725 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:23:12.418847 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:12.418759 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:23:14.490558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:14.490510 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg"] Oct 30 01:23:19.491777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:19.491732 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" podUID="db19fe18-fec2-4511-951b-d612232bda2f" containerName="payload-processing" containerID="cri-o://94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1" gracePeriod=25 Oct 30 01:23:19.832696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:19.832670 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:23:19.895904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:19.895873 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db19fe18-fec2-4511-951b-d612232bda2f-kube-api-access-2fzx2\" (UniqueName: \"kubernetes.io/projected/db19fe18-fec2-4511-951b-d612232bda2f-kube-api-access-2fzx2\") pod \"db19fe18-fec2-4511-951b-d612232bda2f\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " Oct 30 01:23:19.896201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:19.896183 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db19fe18-fec2-4511-951b-d612232bda2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db19fe18-fec2-4511-951b-d612232bda2f-plugins-config-volume\") pod \"db19fe18-fec2-4511-951b-d612232bda2f\" (UID: \"db19fe18-fec2-4511-951b-d612232bda2f\") " Oct 30 01:23:19.896746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:19.896716 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db19fe18-fec2-4511-951b-d612232bda2f-plugins-config-volume" pod "db19fe18-fec2-4511-951b-d612232bda2f" (UID: "db19fe18-fec2-4511-951b-d612232bda2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:19.897188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:19.897156 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db19fe18-fec2-4511-951b-d612232bda2f-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:19.899727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:19.899700 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db19fe18-fec2-4511-951b-d612232bda2f-kube-api-access-2fzx2" pod "db19fe18-fec2-4511-951b-d612232bda2f" (UID: "db19fe18-fec2-4511-951b-d612232bda2f"). InnerVolumeSpecName "kube-api-access-2fzx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:20.003503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.003468 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzx2\" (UniqueName: \"kubernetes.io/projected/db19fe18-fec2-4511-951b-d612232bda2f-kube-api-access-2fzx2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:20.143930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.143837 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc"] Oct 30 01:23:20.144194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.144162 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" podUID="e213edcc-ae82-4ab7-8220-81965e6e6c52" containerName="payload-processing" containerID="cri-o://0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d" gracePeriod=30 Oct 30 01:23:20.147608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.147507 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" event={"ID":"db19fe18-fec2-4511-951b-d612232bda2f","Type":"ContainerDied","Data":"94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1"} Oct 30 01:23:20.147935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.147642 4905 scope.go:122] "RemoveContainer" containerID="94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1" Oct 30 01:23:20.147935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.147650 4905 generic.go:356] "Generic (PLEG): container finished" podID="db19fe18-fec2-4511-951b-d612232bda2f" containerID="94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1" exitCode=0 Oct 30 01:23:20.147935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.147822 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" Oct 30 01:23:20.147935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.147831 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg" event={"ID":"db19fe18-fec2-4511-951b-d612232bda2f","Type":"ContainerDied","Data":"377a50ccf4fe485d0cd4f57b3ec04e2e6e0e580d8844a3096ab40fd2f87da07f"} Oct 30 01:23:20.167042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.167014 4905 scope.go:122] "RemoveContainer" containerID="94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1" Oct 30 01:23:20.167417 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:20.167388 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1\": container with ID starting with 94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1 not found: ID does not exist" containerID="94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1" Oct 30 01:23:20.167617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.167427 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1"} err="failed to get container status \"94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1\": rpc error: code = NotFound desc = could not find container \"94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1\": container with ID starting with 94c19d64a85cd59240e297560a705982c17e3081819e13369326643ce9cf38a1 not found: ID does not exist" Oct 30 01:23:20.283751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.283713 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg"] Oct 30 01:23:20.298282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.297712 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96bd13-66b8d4fcb-8mmsg"] Oct 30 01:23:20.848260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.847824 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:23:20.890789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.890708 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e213edcc-ae82-4ab7-8220-81965e6e6c52-kube-api-access-wmdf7\" (UniqueName: \"kubernetes.io/projected/e213edcc-ae82-4ab7-8220-81965e6e6c52-kube-api-access-wmdf7\") pod \"e213edcc-ae82-4ab7-8220-81965e6e6c52\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " Oct 30 01:23:20.891002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.890979 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e213edcc-ae82-4ab7-8220-81965e6e6c52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e213edcc-ae82-4ab7-8220-81965e6e6c52-plugins-config-volume\") pod \"e213edcc-ae82-4ab7-8220-81965e6e6c52\" (UID: \"e213edcc-ae82-4ab7-8220-81965e6e6c52\") " Oct 30 01:23:20.895151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.894324 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e213edcc-ae82-4ab7-8220-81965e6e6c52-plugins-config-volume" pod "e213edcc-ae82-4ab7-8220-81965e6e6c52" (UID: "e213edcc-ae82-4ab7-8220-81965e6e6c52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:20.930555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:20.924107 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e213edcc-ae82-4ab7-8220-81965e6e6c52-kube-api-access-wmdf7" pod "e213edcc-ae82-4ab7-8220-81965e6e6c52" (UID: "e213edcc-ae82-4ab7-8220-81965e6e6c52"). InnerVolumeSpecName "kube-api-access-wmdf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:21.011556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.006188 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdf7\" (UniqueName: \"kubernetes.io/projected/e213edcc-ae82-4ab7-8220-81965e6e6c52-kube-api-access-wmdf7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.011556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.006228 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e213edcc-ae82-4ab7-8220-81965e6e6c52-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.159136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.159049 4905 generic.go:356] "Generic (PLEG): container finished" podID="e213edcc-ae82-4ab7-8220-81965e6e6c52" containerID="0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d" exitCode=0 Oct 30 01:23:21.159136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.159103 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" Oct 30 01:23:21.159380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.159137 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" event={"ID":"e213edcc-ae82-4ab7-8220-81965e6e6c52","Type":"ContainerDied","Data":"0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d"} Oct 30 01:23:21.159380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.159187 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc" event={"ID":"e213edcc-ae82-4ab7-8220-81965e6e6c52","Type":"ContainerDied","Data":"88c813dfe499794f7d93f04fa8be2a4a2168a21b19b57b4617e6365aa3530607"} Oct 30 01:23:21.159380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.159213 4905 scope.go:122] "RemoveContainer" containerID="0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d" Oct 30 01:23:21.173012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.172992 4905 scope.go:122] "RemoveContainer" containerID="0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d" Oct 30 01:23:21.173294 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:21.173274 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d\": container with ID starting with 0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d not found: ID does not exist" containerID="0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d" Oct 30 01:23:21.173375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.173308 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d"} err="failed to get container status \"0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d\": rpc error: code = NotFound desc = could not find container \"0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d\": container with ID starting with 0ae8360007eae27276207807f8b23c28c698b20423ee6c323d11838682f64a1d not found: ID does not exist" Oct 30 01:23:21.196582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.196554 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db19fe18-fec2-4511-951b-d612232bda2f" path="/var/lib/kubelet/pods/db19fe18-fec2-4511-951b-d612232bda2f/volumes" Oct 30 01:23:21.217804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.217768 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc"] Oct 30 01:23:21.230891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:21.230864 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7bbdfd74f8-gqtlc"] Oct 30 01:23:22.136817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:22.136780 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5"] Oct 30 01:23:22.137480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:22.137039 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" containerID="cri-o://e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186" gracePeriod=30 Oct 30 01:23:23.215069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:23.215031 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e213edcc-ae82-4ab7-8220-81965e6e6c52" path="/var/lib/kubelet/pods/e213edcc-ae82-4ab7-8220-81965e6e6c52/volumes" Oct 30 01:23:27.513559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.513485 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:23:27.625192 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.625156 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-envoy\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.625463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.625227 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-kube-api-access-blbfc\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-kube-api-access-blbfc\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.625463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.625450 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-data\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.625886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.625850 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-data" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:27.626251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.626232 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istiod-ca-cert\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.626384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.626330 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-credential-socket\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.626384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.626372 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-podinfo\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.626591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.626419 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-certs\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.626591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.626450 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-token\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-token\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.626591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.626504 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-socket\") pod \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\" (UID: \"52df6f9a-b866-4fa6-8b50-16606cb8e1ac\") " Oct 30 01:23:27.627379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.627350 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-certs" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:27.627645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.627578 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-socket" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:27.627645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.627593 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.627645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.627612 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.627990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.627788 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-credential-socket" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:27.628240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.628215 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istiod-ca-cert" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:27.629892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.629862 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-kube-api-access-blbfc" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "kube-api-access-blbfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:27.630673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.630652 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-token" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:27.631332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.631308 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-podinfo" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:27.634036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.633961 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-envoy" pod "52df6f9a-b866-4fa6-8b50-16606cb8e1ac" (UID: "52df6f9a-b866-4fa6-8b50-16606cb8e1ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:27.691022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.690943 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf"] Oct 30 01:23:27.691303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.691261 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" containerID="cri-o://8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06" gracePeriod=30 Oct 30 01:23:27.729911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.729871 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.729911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.729911 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.730178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.729926 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blbfc\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-kube-api-access-blbfc\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.730178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.729942 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.730178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.729954 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.730178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.729966 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:27.730178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:27.729979 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52df6f9a-b866-4fa6-8b50-16606cb8e1ac-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.268081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.268042 4905 generic.go:356] "Generic (PLEG): container finished" podID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerID="e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186" exitCode=0 Oct 30 01:23:28.268397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.268126 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" Oct 30 01:23:28.268562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.268140 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" event={"ID":"52df6f9a-b866-4fa6-8b50-16606cb8e1ac","Type":"ContainerDied","Data":"e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186"} Oct 30 01:23:28.268562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.268537 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5" event={"ID":"52df6f9a-b866-4fa6-8b50-16606cb8e1ac","Type":"ContainerDied","Data":"aba875438e63146037091f78e815b94996961f110a2db1f56c72b250200d46c4"} Oct 30 01:23:28.268704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.268574 4905 scope.go:122] "RemoveContainer" containerID="e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186" Oct 30 01:23:28.286387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.286357 4905 scope.go:122] "RemoveContainer" containerID="a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a" Oct 30 01:23:28.304811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.304783 4905 scope.go:122] "RemoveContainer" containerID="e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186" Oct 30 01:23:28.305226 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:28.305200 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186\": container with ID starting with e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186 not found: ID does not exist" containerID="e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186" Oct 30 01:23:28.305348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.305234 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186"} err="failed to get container status \"e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186\": rpc error: code = NotFound desc = could not find container \"e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186\": container with ID starting with e40b733abd934f6ae09c53f511bb18085504b9ad30dc76b4a6312db6c6d5d186 not found: ID does not exist" Oct 30 01:23:28.305348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.305257 4905 scope.go:122] "RemoveContainer" containerID="a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a" Oct 30 01:23:28.305643 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:28.305622 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a\": container with ID starting with a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a not found: ID does not exist" containerID="a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a" Oct 30 01:23:28.305708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.305648 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a"} err="failed to get container status \"a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a\": rpc error: code = NotFound desc = could not find container \"a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a\": container with ID starting with a28370f4405547709c3f6ad9835daed40611051c5224d30caf06df891eea6c4a not found: ID does not exist" Oct 30 01:23:28.328906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.328860 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5"] Oct 30 01:23:28.345621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:28.345587 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f461c3-openshift-default-5758475449-jxrp5"] Oct 30 01:23:29.194653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:29.194621 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" path="/var/lib/kubelet/pods/52df6f9a-b866-4fa6-8b50-16606cb8e1ac/volumes" Oct 30 01:23:33.010910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.010876 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:23:33.148040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.147998 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6a1e07-822c-443b-bf43-293a1455bc8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a1e07-822c-443b-bf43-293a1455bc8b-istiod-ca-cert\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148133 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-envoy\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148188 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-podinfo\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148221 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-socket\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148278 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-kube-api-access-k5zll\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-kube-api-access-k5zll\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148375 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-data\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148415 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6a1e07-822c-443b-bf43-293a1455bc8b-istiod-ca-cert" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:33.148790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148460 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-token\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148514 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-certs\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.148790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.148737 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-credential-socket\") pod \"0c6a1e07-822c-443b-bf43-293a1455bc8b\" (UID: \"0c6a1e07-822c-443b-bf43-293a1455bc8b\") " Oct 30 01:23:33.149264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.149241 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-socket" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.149649 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.149447 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-certs" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.149823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.149791 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-credential-socket" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.150028 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.150005 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-data" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.150543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.150411 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.150543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.150439 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.150543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.150456 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a1e07-822c-443b-bf43-293a1455bc8b-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.150543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.150472 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.150543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.150488 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.151393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.151363 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-envoy" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.151897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.151872 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-kube-api-access-k5zll" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "kube-api-access-k5zll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:33.152307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.152287 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-token" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:33.152966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.152947 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-podinfo" pod "0c6a1e07-822c-443b-bf43-293a1455bc8b" (UID: "0c6a1e07-822c-443b-bf43-293a1455bc8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:33.252568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.252534 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.252568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.252565 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.252568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.252575 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5zll\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-kube-api-access-k5zll\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.252831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.252586 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a1e07-822c-443b-bf43-293a1455bc8b-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.320145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.320109 4905 generic.go:356] "Generic (PLEG): container finished" podID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerID="8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06" exitCode=0 Oct 30 01:23:33.320341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.320158 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" Oct 30 01:23:33.320341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.320189 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" event={"ID":"0c6a1e07-822c-443b-bf43-293a1455bc8b","Type":"ContainerDied","Data":"8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06"} Oct 30 01:23:33.320341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.320247 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf" event={"ID":"0c6a1e07-822c-443b-bf43-293a1455bc8b","Type":"ContainerDied","Data":"b9f9043a1f0059c6ef801b40fb29a9018668f4fd5b021496840777c7de852d25"} Oct 30 01:23:33.320341 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.320277 4905 scope.go:122] "RemoveContainer" containerID="8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06" Oct 30 01:23:33.332744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.332722 4905 scope.go:122] "RemoveContainer" containerID="9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496" Oct 30 01:23:33.347130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.347100 4905 scope.go:122] "RemoveContainer" containerID="8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06" Oct 30 01:23:33.347447 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:33.347424 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06\": container with ID starting with 8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06 not found: ID does not exist" containerID="8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06" Oct 30 01:23:33.347505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.347456 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06"} err="failed to get container status \"8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06\": rpc error: code = NotFound desc = could not find container \"8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06\": container with ID starting with 8085218dc76b8d7ff2525ce1426fdb25a685eb0cc86acdc67dbfe152c1c03c06 not found: ID does not exist" Oct 30 01:23:33.347505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.347474 4905 scope.go:122] "RemoveContainer" containerID="9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496" Oct 30 01:23:33.348041 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:33.348021 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496\": container with ID starting with 9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496 not found: ID does not exist" containerID="9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496" Oct 30 01:23:33.348118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.348050 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496"} err="failed to get container status \"9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496\": rpc error: code = NotFound desc = could not find container \"9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496\": container with ID starting with 9e3ef57b07fec6af49833f0423f1f43e224ef887645c7ad04d3df9eeac62d496 not found: ID does not exist" Oct 30 01:23:33.380489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.380457 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf"] Oct 30 01:23:33.393264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:33.393229 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1075cb-openshift-default-75694fdbc9-qsdcf"] Oct 30 01:23:34.042733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.042699 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4"] Oct 30 01:23:34.043144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.042879 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" podUID="3f36f245-3b68-4918-8109-d80b64f270a8" containerName="payload-processing" containerID="cri-o://ef0d62b5a365b93a2429d62efc81813c13b902147d16e359e6df69c476491af9" gracePeriod=30 Oct 30 01:23:34.059122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.059092 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh"] Oct 30 01:23:34.059360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.059316 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" podUID="5b0c33c3-8280-406c-ae34-b032c03b8432" containerName="payload-pre-processing" containerID="cri-o://6705f580716c0a6ac0f35db7bc59873f91aa7cbcc3fc21931c85e4d19c9834f3" gracePeriod=30 Oct 30 01:23:34.369741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.369700 4905 generic.go:356] "Generic (PLEG): container finished" podID="5b0c33c3-8280-406c-ae34-b032c03b8432" containerID="6705f580716c0a6ac0f35db7bc59873f91aa7cbcc3fc21931c85e4d19c9834f3" exitCode=0 Oct 30 01:23:34.370098 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.369955 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" event={"ID":"5b0c33c3-8280-406c-ae34-b032c03b8432","Type":"ContainerDied","Data":"6705f580716c0a6ac0f35db7bc59873f91aa7cbcc3fc21931c85e4d19c9834f3"} Oct 30 01:23:34.379163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.377835 4905 generic.go:356] "Generic (PLEG): container finished" podID="3f36f245-3b68-4918-8109-d80b64f270a8" containerID="ef0d62b5a365b93a2429d62efc81813c13b902147d16e359e6df69c476491af9" exitCode=0 Oct 30 01:23:34.379163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.377945 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" event={"ID":"3f36f245-3b68-4918-8109-d80b64f270a8","Type":"ContainerDied","Data":"ef0d62b5a365b93a2429d62efc81813c13b902147d16e359e6df69c476491af9"} Oct 30 01:23:34.455703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.451931 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:23:34.461290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.461263 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97"] Oct 30 01:23:34.461493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.461472 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" podUID="8eb0e12d-c66f-43e3-a763-8a001b1b12a0" containerName="payload-processing" containerID="cri-o://63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c" gracePeriod=30 Oct 30 01:23:34.491073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.491036 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f36f245-3b68-4918-8109-d80b64f270a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36f245-3b68-4918-8109-d80b64f270a8-plugins-config-volume\") pod \"3f36f245-3b68-4918-8109-d80b64f270a8\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " Oct 30 01:23:34.491279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.491256 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f36f245-3b68-4918-8109-d80b64f270a8-kube-api-access-6bj4b\" (UniqueName: \"kubernetes.io/projected/3f36f245-3b68-4918-8109-d80b64f270a8-kube-api-access-6bj4b\") pod \"3f36f245-3b68-4918-8109-d80b64f270a8\" (UID: \"3f36f245-3b68-4918-8109-d80b64f270a8\") " Oct 30 01:23:34.492788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.492762 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f36f245-3b68-4918-8109-d80b64f270a8-plugins-config-volume" pod "3f36f245-3b68-4918-8109-d80b64f270a8" (UID: "3f36f245-3b68-4918-8109-d80b64f270a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:34.495577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.495554 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f36f245-3b68-4918-8109-d80b64f270a8-kube-api-access-6bj4b" pod "3f36f245-3b68-4918-8109-d80b64f270a8" (UID: "3f36f245-3b68-4918-8109-d80b64f270a8"). InnerVolumeSpecName "kube-api-access-6bj4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:34.495821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.495792 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp"] Oct 30 01:23:34.496101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.496049 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" podUID="248f81ea-269e-492e-abc0-0c8194898606" containerName="payload-pre-processing" containerID="cri-o://cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7" gracePeriod=30 Oct 30 01:23:34.593258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.593229 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bj4b\" (UniqueName: \"kubernetes.io/projected/3f36f245-3b68-4918-8109-d80b64f270a8-kube-api-access-6bj4b\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:34.593258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.593257 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36f245-3b68-4918-8109-d80b64f270a8-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:34.639065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.639028 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:23:34.694393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.694343 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0c33c3-8280-406c-ae34-b032c03b8432-kube-api-access-49pv5\" (UniqueName: \"kubernetes.io/projected/5b0c33c3-8280-406c-ae34-b032c03b8432-kube-api-access-49pv5\") pod \"5b0c33c3-8280-406c-ae34-b032c03b8432\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " Oct 30 01:23:34.694777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.694749 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0c33c3-8280-406c-ae34-b032c03b8432-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c33c3-8280-406c-ae34-b032c03b8432-plugins-config-volume\") pod \"5b0c33c3-8280-406c-ae34-b032c03b8432\" (UID: \"5b0c33c3-8280-406c-ae34-b032c03b8432\") " Oct 30 01:23:34.695974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.695947 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0c33c3-8280-406c-ae34-b032c03b8432-plugins-config-volume" pod "5b0c33c3-8280-406c-ae34-b032c03b8432" (UID: "5b0c33c3-8280-406c-ae34-b032c03b8432"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:34.698807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.698783 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0c33c3-8280-406c-ae34-b032c03b8432-kube-api-access-49pv5" pod "5b0c33c3-8280-406c-ae34-b032c03b8432" (UID: "5b0c33c3-8280-406c-ae34-b032c03b8432"). InnerVolumeSpecName "kube-api-access-49pv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:34.831604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.822947 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49pv5\" (UniqueName: \"kubernetes.io/projected/5b0c33c3-8280-406c-ae34-b032c03b8432-kube-api-access-49pv5\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:34.831604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:34.822985 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0c33c3-8280-406c-ae34-b032c03b8432-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:35.216246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.216208 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" path="/var/lib/kubelet/pods/0c6a1e07-822c-443b-bf43-293a1455bc8b/volumes" Oct 30 01:23:35.320565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.317841 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:23:35.400741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.400694 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" event={"ID":"5b0c33c3-8280-406c-ae34-b032c03b8432","Type":"ContainerDied","Data":"966c013b4bbc03051e3b79e4eb8087f77d05eec8dd8a38512e39522c5f909bb2"} Oct 30 01:23:35.400935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.400802 4905 scope.go:122] "RemoveContainer" containerID="6705f580716c0a6ac0f35db7bc59873f91aa7cbcc3fc21931c85e4d19c9834f3" Oct 30 01:23:35.401017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.400960 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh" Oct 30 01:23:35.402252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.402066 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:23:35.403959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.403931 4905 generic.go:356] "Generic (PLEG): container finished" podID="248f81ea-269e-492e-abc0-0c8194898606" containerID="cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7" exitCode=0 Oct 30 01:23:35.404078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.404050 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" event={"ID":"248f81ea-269e-492e-abc0-0c8194898606","Type":"ContainerDied","Data":"cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7"} Oct 30 01:23:35.404156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.404091 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" event={"ID":"248f81ea-269e-492e-abc0-0c8194898606","Type":"ContainerDied","Data":"a05554c1583b21e9bbf4dea6a4e4d7130b8b6128dc7d885b8314912f78b54581"} Oct 30 01:23:35.405982 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.405958 4905 generic.go:356] "Generic (PLEG): container finished" podID="8eb0e12d-c66f-43e3-a763-8a001b1b12a0" containerID="63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c" exitCode=0 Oct 30 01:23:35.406105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.406038 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" Oct 30 01:23:35.406174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.406136 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" event={"ID":"8eb0e12d-c66f-43e3-a763-8a001b1b12a0","Type":"ContainerDied","Data":"63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c"} Oct 30 01:23:35.406241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.406174 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97" event={"ID":"8eb0e12d-c66f-43e3-a763-8a001b1b12a0","Type":"ContainerDied","Data":"6f28bf0792649b863a25d79a14b7e3d00ba6eec4222b444d9ec5e0512d0717f9"} Oct 30 01:23:35.408399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.408375 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" event={"ID":"3f36f245-3b68-4918-8109-d80b64f270a8","Type":"ContainerDied","Data":"b23a32bfdde482165467b14d00ecdd1c2ca32ee65a1e67d1d9ca8f579ab76e3e"} Oct 30 01:23:35.408573 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.408556 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4" Oct 30 01:23:35.415085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.414287 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-plugins-config-volume\") pod \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " Oct 30 01:23:35.415085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.414677 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-kube-api-access-s846j\" (UniqueName: \"kubernetes.io/projected/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-kube-api-access-s846j\") pod \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\" (UID: \"8eb0e12d-c66f-43e3-a763-8a001b1b12a0\") " Oct 30 01:23:35.415085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.415019 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-plugins-config-volume" pod "8eb0e12d-c66f-43e3-a763-8a001b1b12a0" (UID: "8eb0e12d-c66f-43e3-a763-8a001b1b12a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:35.415832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.415649 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:35.417271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.417246 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-kube-api-access-s846j" pod "8eb0e12d-c66f-43e3-a763-8a001b1b12a0" (UID: "8eb0e12d-c66f-43e3-a763-8a001b1b12a0"). InnerVolumeSpecName "kube-api-access-s846j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:35.424161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.423941 4905 scope.go:122] "RemoveContainer" containerID="cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7" Oct 30 01:23:35.445281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.445241 4905 scope.go:122] "RemoveContainer" containerID="cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7" Oct 30 01:23:35.445661 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:35.445634 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7\": container with ID starting with cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7 not found: ID does not exist" containerID="cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7" Oct 30 01:23:35.445840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.445673 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7"} err="failed to get container status \"cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7\": rpc error: code = NotFound desc = could not find container \"cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7\": container with ID starting with cb36074e63d04ca0b7015a09fcb99a4115b3c93d17ad7b179278b5673a10fdc7 not found: ID does not exist" Oct 30 01:23:35.445840 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.445699 4905 scope.go:122] "RemoveContainer" containerID="63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c" Oct 30 01:23:35.458151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.458133 4905 scope.go:122] "RemoveContainer" containerID="63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c" Oct 30 01:23:35.458498 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:35.458474 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c\": container with ID starting with 63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c not found: ID does not exist" containerID="63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c" Oct 30 01:23:35.458605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.458507 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c"} err="failed to get container status \"63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c\": rpc error: code = NotFound desc = could not find container \"63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c\": container with ID starting with 63372fc8841d23384ac05b229f7a88dcb90cd40172e74c7e09be04b0edd8353c not found: ID does not exist" Oct 30 01:23:35.458605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.458575 4905 scope.go:122] "RemoveContainer" containerID="ef0d62b5a365b93a2429d62efc81813c13b902147d16e359e6df69c476491af9" Oct 30 01:23:35.460133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.460110 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4"] Oct 30 01:23:35.464921 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.464875 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-685b46b84d-ksrl4"] Oct 30 01:23:35.511753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.511719 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh"] Oct 30 01:23:35.515828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.515801 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-9bc7b48f5-jktgh"] Oct 30 01:23:35.516257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.516236 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248f81ea-269e-492e-abc0-0c8194898606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248f81ea-269e-492e-abc0-0c8194898606-plugins-config-volume\") pod \"248f81ea-269e-492e-abc0-0c8194898606\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " Oct 30 01:23:35.516468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.516453 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248f81ea-269e-492e-abc0-0c8194898606-kube-api-access-wq66r\" (UniqueName: \"kubernetes.io/projected/248f81ea-269e-492e-abc0-0c8194898606-kube-api-access-wq66r\") pod \"248f81ea-269e-492e-abc0-0c8194898606\" (UID: \"248f81ea-269e-492e-abc0-0c8194898606\") " Oct 30 01:23:35.516619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.516599 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248f81ea-269e-492e-abc0-0c8194898606-plugins-config-volume" pod "248f81ea-269e-492e-abc0-0c8194898606" (UID: "248f81ea-269e-492e-abc0-0c8194898606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:35.517186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.517152 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s846j\" (UniqueName: \"kubernetes.io/projected/8eb0e12d-c66f-43e3-a763-8a001b1b12a0-kube-api-access-s846j\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:35.517260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.517190 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248f81ea-269e-492e-abc0-0c8194898606-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:35.518812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.518789 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248f81ea-269e-492e-abc0-0c8194898606-kube-api-access-wq66r" pod "248f81ea-269e-492e-abc0-0c8194898606" (UID: "248f81ea-269e-492e-abc0-0c8194898606"). InnerVolumeSpecName "kube-api-access-wq66r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:35.628210 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.624302 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq66r\" (UniqueName: \"kubernetes.io/projected/248f81ea-269e-492e-abc0-0c8194898606-kube-api-access-wq66r\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:35.758641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.758600 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97"] Oct 30 01:23:35.764262 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:35.764230 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-999dfd56b-9sk97"] Oct 30 01:23:36.417780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:36.417702 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp" Oct 30 01:23:36.462287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:36.462245 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496"] Oct 30 01:23:36.474647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:36.474600 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:23:36.476888 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:36.476863 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:23:36.477334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:36.477310 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp"] Oct 30 01:23:36.482534 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:36.482496 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c515e-f6fd7d5f-lpjjp"] Oct 30 01:23:37.195465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.195429 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248f81ea-269e-492e-abc0-0c8194898606" path="/var/lib/kubelet/pods/248f81ea-269e-492e-abc0-0c8194898606/volumes" Oct 30 01:23:37.196125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.196102 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f36f245-3b68-4918-8109-d80b64f270a8" path="/var/lib/kubelet/pods/3f36f245-3b68-4918-8109-d80b64f270a8/volumes" Oct 30 01:23:37.196654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.196633 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0c33c3-8280-406c-ae34-b032c03b8432" path="/var/lib/kubelet/pods/5b0c33c3-8280-406c-ae34-b032c03b8432/volumes" Oct 30 01:23:37.197169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.197150 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb0e12d-c66f-43e3-a763-8a001b1b12a0" path="/var/lib/kubelet/pods/8eb0e12d-c66f-43e3-a763-8a001b1b12a0/volumes" Oct 30 01:23:37.717542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.712893 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p"] Oct 30 01:23:37.735924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.735888 4905 cpu_manager.go:414] "containerMap: removing container" podUID="248f81ea-269e-492e-abc0-0c8194898606" containerName="payload-pre-processing" Oct 30 01:23:37.736159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.736143 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248f81ea-269e-492e-abc0-0c8194898606" containerName="payload-pre-processing" Oct 30 01:23:37.736276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.736266 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8eb0e12d-c66f-43e3-a763-8a001b1b12a0" containerName="payload-processing" Oct 30 01:23:37.736360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.736351 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb0e12d-c66f-43e3-a763-8a001b1b12a0" containerName="payload-processing" Oct 30 01:23:37.736536 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.736458 4905 cpu_manager.go:414] "containerMap: removing container" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" Oct 30 01:23:37.736729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.736717 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" Oct 30 01:23:37.736823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.736814 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" Oct 30 01:23:37.736908 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.736899 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" Oct 30 01:23:37.737071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.737062 4905 cpu_manager.go:414] "containerMap: removing container" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" Oct 30 01:23:37.737158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.737147 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" Oct 30 01:23:37.737265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.737255 4905 cpu_manager.go:414] "containerMap: removing container" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" Oct 30 01:23:37.737404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.737395 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" Oct 30 01:23:37.737565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.737554 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5b0c33c3-8280-406c-ae34-b032c03b8432" containerName="payload-pre-processing" Oct 30 01:23:37.737748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.737737 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0c33c3-8280-406c-ae34-b032c03b8432" containerName="payload-pre-processing" Oct 30 01:23:37.737940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.737927 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3f36f245-3b68-4918-8109-d80b64f270a8" containerName="payload-processing" Oct 30 01:23:37.738129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.738100 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f36f245-3b68-4918-8109-d80b64f270a8" containerName="payload-processing" Oct 30 01:23:37.738249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.738239 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" Oct 30 01:23:37.739204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.739185 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" Oct 30 01:23:37.739373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.739359 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" Oct 30 01:23:37.739641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.739627 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" Oct 30 01:23:37.739902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.739888 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" Oct 30 01:23:37.740118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.740108 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" Oct 30 01:23:37.740253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.740241 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e213edcc-ae82-4ab7-8220-81965e6e6c52" containerName="payload-processing" Oct 30 01:23:37.740951 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.740930 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e213edcc-ae82-4ab7-8220-81965e6e6c52" containerName="payload-processing" Oct 30 01:23:37.741136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.741123 4905 cpu_manager.go:414] "containerMap: removing container" podUID="db19fe18-fec2-4511-951b-d612232bda2f" containerName="payload-processing" Oct 30 01:23:37.741249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.741237 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db19fe18-fec2-4511-951b-d612232bda2f" containerName="payload-processing" Oct 30 01:23:37.741863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.741821 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" Oct 30 01:23:37.742295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.742282 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e213edcc-ae82-4ab7-8220-81965e6e6c52" containerName="payload-processing" Oct 30 01:23:37.742480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.742470 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="db19fe18-fec2-4511-951b-d612232bda2f" containerName="payload-processing" Oct 30 01:23:37.742591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.742580 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="248f81ea-269e-492e-abc0-0c8194898606" containerName="payload-pre-processing" Oct 30 01:23:37.742697 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.742687 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb0e12d-c66f-43e3-a763-8a001b1b12a0" containerName="payload-processing" Oct 30 01:23:37.742846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.742836 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" Oct 30 01:23:37.742989 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.742979 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f19f52-412e-4c09-8323-6fed84a9a68b" containerName="main" Oct 30 01:23:37.744944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.744925 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d62e24-c720-4e25-a170-a5740a38ee8f" containerName="main" Oct 30 01:23:37.745157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.745146 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="52df6f9a-b866-4fa6-8b50-16606cb8e1ac" containerName="istio-proxy" Oct 30 01:23:37.745272 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.745261 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0c33c3-8280-406c-ae34-b032c03b8432" containerName="payload-pre-processing" Oct 30 01:23:37.745485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.745472 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f36f245-3b68-4918-8109-d80b64f270a8" containerName="payload-processing" Oct 30 01:23:37.745721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.745707 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" Oct 30 01:23:37.745855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.745843 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" Oct 30 01:23:37.746058 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.746046 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e37aa37-d267-4038-93ff-6510ffd77977" containerName="istio-proxy" Oct 30 01:23:37.757172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.757146 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:37.759374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.759352 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p"] Oct 30 01:23:37.816602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.801229 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-11fbe8\"" Oct 30 01:23:37.816602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.801365 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-11fbe8-dockercfg-ngwl2\"" Oct 30 01:23:37.931399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.925859 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aad8a0-1157-439a-84f7-7ca03ab693b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:37.931399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.925933 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbs2\" (UniqueName: \"kubernetes.io/projected/58aad8a0-1157-439a-84f7-7ca03ab693b1-kube-api-access-cdbs2\") pod \"payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:37.981435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.980481 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv"] Oct 30 01:23:37.983964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.983936 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" Oct 30 01:23:37.984143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.984127 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a1e07-822c-443b-bf43-293a1455bc8b" containerName="istio-proxy" Oct 30 01:23:37.984252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.984240 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" Oct 30 01:23:37.984348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.984337 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" Oct 30 01:23:37.984948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.984926 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61f6648-27e9-448f-8e07-e2ed4341a36a" containerName="istio-proxy" Oct 30 01:23:37.989897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.989878 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv"] Oct 30 01:23:37.990093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:37.990002 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.027432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.027392 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aad8a0-1157-439a-84f7-7ca03ab693b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:38.027622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.027437 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbs2\" (UniqueName: \"kubernetes.io/projected/58aad8a0-1157-439a-84f7-7ca03ab693b1-kube-api-access-cdbs2\") pod \"payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:38.029468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.029442 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aad8a0-1157-439a-84f7-7ca03ab693b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:38.051480 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.051443 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbs2\" (UniqueName: \"kubernetes.io/projected/58aad8a0-1157-439a-84f7-7ca03ab693b1-kube-api-access-cdbs2\") pod \"payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:38.129831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.129787 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zfg\" (UniqueName: \"kubernetes.io/projected/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-kube-api-access-m2zfg\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.130102 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.130078 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.232361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.232257 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zfg\" (UniqueName: \"kubernetes.io/projected/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-kube-api-access-m2zfg\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.232361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.232312 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.233587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.233552 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.254693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.254666 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zfg\" (UniqueName: \"kubernetes.io/projected/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-kube-api-access-m2zfg\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.255421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.255404 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:38.319106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.318874 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:38.583878 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:23:38.583849 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aad8a0_1157_439a_84f7_7ca03ab693b1.slice/crio-b7f032db2844d7d3a50d25379cda72b23c2d75506c942a5a2552aeb9064a8354 WatchSource:0}: Error finding container b7f032db2844d7d3a50d25379cda72b23c2d75506c942a5a2552aeb9064a8354: Status 404 returned error can't find the container with id b7f032db2844d7d3a50d25379cda72b23c2d75506c942a5a2552aeb9064a8354 Oct 30 01:23:38.589499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.589422 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p"] Oct 30 01:23:38.652260 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:23:38.652214 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f76eb2_bf0e_4ffe_8653_d5596d1b2037.slice/crio-0197322f51d41682e699e9f1dbf47b5aca86a2c714bc53b3c07fd40dd3df8a51 WatchSource:0}: Error finding container 0197322f51d41682e699e9f1dbf47b5aca86a2c714bc53b3c07fd40dd3df8a51: Status 404 returned error can't find the container with id 0197322f51d41682e699e9f1dbf47b5aca86a2c714bc53b3c07fd40dd3df8a51 Oct 30 01:23:38.653602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:38.653579 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv"] Oct 30 01:23:39.568565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.568252 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" event={"ID":"58aad8a0-1157-439a-84f7-7ca03ab693b1","Type":"ContainerStarted","Data":"51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568"} Oct 30 01:23:39.568565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.568325 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:39.568565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.568349 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" event={"ID":"58aad8a0-1157-439a-84f7-7ca03ab693b1","Type":"ContainerStarted","Data":"b7f032db2844d7d3a50d25379cda72b23c2d75506c942a5a2552aeb9064a8354"} Oct 30 01:23:39.570435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.570399 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" event={"ID":"63f76eb2-bf0e-4ffe-8653-d5596d1b2037","Type":"ContainerStarted","Data":"705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3"} Oct 30 01:23:39.570586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.570463 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" event={"ID":"63f76eb2-bf0e-4ffe-8653-d5596d1b2037","Type":"ContainerStarted","Data":"0197322f51d41682e699e9f1dbf47b5aca86a2c714bc53b3c07fd40dd3df8a51"} Oct 30 01:23:39.570586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.570493 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:39.614430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.614363 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" podStartSLOduration=2.614344842 podStartE2EDuration="2.614344842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:39.609671528 +0000 UTC m=+2704.552844721" watchObservedRunningTime="2026-10-30 01:23:39.614344842 +0000 UTC m=+2704.557518037" Oct 30 01:23:39.654986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:39.654933 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" podStartSLOduration=2.654918048 podStartE2EDuration="2.654918048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:39.651941972 +0000 UTC m=+2704.595115168" watchObservedRunningTime="2026-10-30 01:23:39.654918048 +0000 UTC m=+2704.598091304" Oct 30 01:23:41.161705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:41.160917 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc"] Oct 30 01:23:41.161705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:41.161165 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" containerID="cri-o://58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a" gracePeriod=30 Oct 30 01:23:41.174030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:41.173995 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m"] Oct 30 01:23:41.174261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:41.174236 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" containerID="cri-o://f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0" gracePeriod=30 Oct 30 01:23:41.466600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:41.463806 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" podUID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" containerName="payload-pre-processing" containerID="cri-o://6d46f348e1fa0758b37f95b5909feea7a5473105dca23945f26d863bd8b06fd1" gracePeriod=25 Oct 30 01:23:42.421497 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:42.421460 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:23:42.523174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:42.523128 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh"] Oct 30 01:23:42.524308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:42.523387 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" containerID="cri-o://a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422" gracePeriod=30 Oct 30 01:23:42.532885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:42.532578 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz"] Oct 30 01:23:42.533339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:42.533307 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" containerID="cri-o://cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344" gracePeriod=30 Oct 30 01:23:42.535443 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:42.535422 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw"] Oct 30 01:23:42.535757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:42.535726 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" containerID="cri-o://8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22" gracePeriod=30 Oct 30 01:23:46.623207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.623174 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:23:46.702696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.702664 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:23:46.742881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.742844 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-podinfo\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.742949 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-token\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-token\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.742982 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istiod-ca-cert\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.743073 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-certs\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.743104 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-kube-api-access-qtxrm\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-kube-api-access-qtxrm\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.743142 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-socket\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.743262 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-data\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.743353 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-credential-socket\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.743449 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-envoy\") pod \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\" (UID: \"5cf94bd3-19e1-4e64-85ca-bda6f576079d\") " Oct 30 01:23:46.743842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.743618 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istiod-ca-cert" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:46.744486 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.744350 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.744712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.744684 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-data" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.744839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.744753 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-certs" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.744839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.744800 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-credential-socket" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.745033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.745015 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-socket" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.765774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.755330 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-envoy" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.765774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.756108 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-podinfo" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:46.765774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.756179 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-token" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:46.765774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.758058 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-kube-api-access-qtxrm" pod "5cf94bd3-19e1-4e64-85ca-bda6f576079d" (UID: "5cf94bd3-19e1-4e64-85ca-bda6f576079d"). InnerVolumeSpecName "kube-api-access-qtxrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:46.845729 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.845691 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-credential-socket\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.845995 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.845752 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-certs\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.845995 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.845943 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istiod-ca-cert\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.846166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846003 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-credential-socket" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.846166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846041 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-token\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-token\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.846166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846136 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-data\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.846426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846199 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-socket\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.846426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846252 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-podinfo\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.846426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846277 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-certs" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.846426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846302 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-envoy\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.846426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846399 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-kube-api-access-szmmx\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-kube-api-access-szmmx\") pod \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\" (UID: \"e9ea456d-9106-466b-b8eb-bea896ccd4b4\") " Oct 30 01:23:46.846794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846482 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-data" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.846854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.846810 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istiod-ca-cert" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847248 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847275 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847289 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847304 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847317 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847330 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtxrm\" (UniqueName: \"kubernetes.io/projected/5cf94bd3-19e1-4e64-85ca-bda6f576079d-kube-api-access-qtxrm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847344 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847357 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847371 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847384 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf94bd3-19e1-4e64-85ca-bda6f576079d-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847399 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.847412 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.848484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.848237 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-socket" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.849796 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.849746 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-kube-api-access-szmmx" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "kube-api-access-szmmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:46.849911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.849867 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-podinfo" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:46.851904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.851879 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-token" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:46.852191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.852169 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-envoy" pod "e9ea456d-9106-466b-b8eb-bea896ccd4b4" (UID: "e9ea456d-9106-466b-b8eb-bea896ccd4b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:46.924822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.924791 4905 generic.go:356] "Generic (PLEG): container finished" podID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerID="f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0" exitCode=0 Oct 30 01:23:46.925012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.924840 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" Oct 30 01:23:46.925012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.924874 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" event={"ID":"e9ea456d-9106-466b-b8eb-bea896ccd4b4","Type":"ContainerDied","Data":"f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0"} Oct 30 01:23:46.925012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.924936 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m" event={"ID":"e9ea456d-9106-466b-b8eb-bea896ccd4b4","Type":"ContainerDied","Data":"b8aeac9037c00c48c6c9f723efb6dfdddcb1c17cbbf81f51489367b0bd561a36"} Oct 30 01:23:46.925012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.924967 4905 scope.go:122] "RemoveContainer" containerID="f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0" Oct 30 01:23:46.926900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.926878 4905 generic.go:356] "Generic (PLEG): container finished" podID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerID="58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a" exitCode=0 Oct 30 01:23:46.927037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.926928 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" Oct 30 01:23:46.927227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.927120 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" event={"ID":"5cf94bd3-19e1-4e64-85ca-bda6f576079d","Type":"ContainerDied","Data":"58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a"} Oct 30 01:23:46.927227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.927164 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc" event={"ID":"5cf94bd3-19e1-4e64-85ca-bda6f576079d","Type":"ContainerDied","Data":"38d3603fb05f20b1bf13b26082acd4d7fc9c9c042b1f5ddb7fce96a95d5cb959"} Oct 30 01:23:46.938710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.938688 4905 scope.go:122] "RemoveContainer" containerID="f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0" Oct 30 01:23:46.939071 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:46.939031 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0\": container with ID starting with f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0 not found: ID does not exist" containerID="f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0" Oct 30 01:23:46.939172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.939079 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0"} err="failed to get container status \"f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0\": rpc error: code = NotFound desc = could not find container \"f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0\": container with ID starting with f68b1a1234537b3263124dbaab4a18fef66ad5fc5fccc1af608e792addfe1bc0 not found: ID does not exist" Oct 30 01:23:46.939172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.939101 4905 scope.go:122] "RemoveContainer" containerID="58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a" Oct 30 01:23:46.949193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.949168 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.949276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.949198 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.949276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.949212 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.949276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.949228 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ea456d-9106-466b-b8eb-bea896ccd4b4-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.949276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.949243 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szmmx\" (UniqueName: \"kubernetes.io/projected/e9ea456d-9106-466b-b8eb-bea896ccd4b4-kube-api-access-szmmx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:46.951117 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.951068 4905 scope.go:122] "RemoveContainer" containerID="58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a" Oct 30 01:23:46.951410 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:46.951372 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a\": container with ID starting with 58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a not found: ID does not exist" containerID="58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a" Oct 30 01:23:46.951491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.951409 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a"} err="failed to get container status \"58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a\": rpc error: code = NotFound desc = could not find container \"58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a\": container with ID starting with 58a414fff08bcbe7bd5e4f25741a3fbd7441757a33047eb052f23d261bfe3d5a not found: ID does not exist" Oct 30 01:23:46.984110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.984063 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m"] Oct 30 01:23:46.990797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:46.990771 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-7gx8m"] Oct 30 01:23:47.027924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.027900 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc"] Oct 30 01:23:47.032004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.031981 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-c2ckc"] Oct 30 01:23:47.198624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.198582 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" path="/var/lib/kubelet/pods/5cf94bd3-19e1-4e64-85ca-bda6f576079d/volumes" Oct 30 01:23:47.199278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.199253 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" path="/var/lib/kubelet/pods/e9ea456d-9106-466b-b8eb-bea896ccd4b4/volumes" Oct 30 01:23:47.889295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.888975 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:23:47.924753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.924587 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:23:47.942216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.941629 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:23:47.944658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.944629 4905 generic.go:356] "Generic (PLEG): container finished" podID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerID="cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344" exitCode=0 Oct 30 01:23:47.944860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.944841 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" event={"ID":"afffce59-7742-4349-abc3-0ab7e9a846a1","Type":"ContainerDied","Data":"cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344"} Oct 30 01:23:47.944949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.944896 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" event={"ID":"afffce59-7742-4349-abc3-0ab7e9a846a1","Type":"ContainerDied","Data":"2d8be7429b5ebd753b5bda587281f652b571f1b172f42248357f6993cc4b3e28"} Oct 30 01:23:47.944949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.944925 4905 scope.go:122] "RemoveContainer" containerID="cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344" Oct 30 01:23:47.945663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.945034 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz" Oct 30 01:23:47.952717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.952694 4905 generic.go:356] "Generic (PLEG): container finished" podID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerID="a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422" exitCode=0 Oct 30 01:23:47.952990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.952844 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" Oct 30 01:23:47.953344 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.952865 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" event={"ID":"9a911e25-7c2d-424c-8a9e-462891ac0508","Type":"ContainerDied","Data":"a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422"} Oct 30 01:23:47.953344 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.953161 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh" event={"ID":"9a911e25-7c2d-424c-8a9e-462891ac0508","Type":"ContainerDied","Data":"115be8145a789b4584f13979922a97db6b307d7173876c74d4351c0e31ac927b"} Oct 30 01:23:47.958951 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.958935 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" Oct 30 01:23:47.959251 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.959228 4905 generic.go:356] "Generic (PLEG): container finished" podID="0244d952-9b69-4cdd-bd71-349849e060a5" containerID="8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22" exitCode=0 Oct 30 01:23:47.959452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.959375 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" event={"ID":"0244d952-9b69-4cdd-bd71-349849e060a5","Type":"ContainerDied","Data":"8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22"} Oct 30 01:23:47.963706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.963685 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw" event={"ID":"0244d952-9b69-4cdd-bd71-349849e060a5","Type":"ContainerDied","Data":"1ca94fcc19adcb71f649e27f3287915455b780f9e6669d736deee480b27de36f"} Oct 30 01:23:47.969344 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.969327 4905 scope.go:122] "RemoveContainer" containerID="cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344" Oct 30 01:23:47.970242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.970218 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-credential-socket\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.970492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.970476 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-token\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-token\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.971046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.971027 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-kube-api-access-kxndg\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-kube-api-access-kxndg\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.971279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.971263 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a911e25-7c2d-424c-8a9e-462891ac0508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a911e25-7c2d-424c-8a9e-462891ac0508-istiod-ca-cert\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.971473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.971456 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-socket\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.971645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.971627 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-data\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.971851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.971824 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-envoy\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.971946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.971934 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-certs\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.972053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.972041 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-podinfo\") pod \"9a911e25-7c2d-424c-8a9e-462891ac0508\" (UID: \"9a911e25-7c2d-424c-8a9e-462891ac0508\") " Oct 30 01:23:47.973400 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:47.973378 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344\": container with ID starting with cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344 not found: ID does not exist" containerID="cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344" Oct 30 01:23:47.973638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.973613 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344"} err="failed to get container status \"cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344\": rpc error: code = NotFound desc = could not find container \"cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344\": container with ID starting with cb143e236e8531d6ff5c2ff75315b228f2c8174a8849707cab9c59f123bfd344 not found: ID does not exist" Oct 30 01:23:47.973758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.973746 4905 scope.go:122] "RemoveContainer" containerID="a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422" Oct 30 01:23:47.975619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.975598 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a911e25-7c2d-424c-8a9e-462891ac0508-istiod-ca-cert" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:47.976026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.976008 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-credential-socket" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:47.977378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.977257 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-data" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:47.977378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.977313 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-certs" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:47.977378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:47.977358 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-socket" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.004453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.004423 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-kube-api-access-kxndg" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "kube-api-access-kxndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:48.005569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.005512 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-envoy" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.006689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.006654 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-token" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:48.009353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.009260 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-podinfo" pod "9a911e25-7c2d-424c-8a9e-462891ac0508" (UID: "9a911e25-7c2d-424c-8a9e-462891ac0508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:48.021217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.020863 4905 scope.go:122] "RemoveContainer" containerID="a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422" Oct 30 01:23:48.021876 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:48.021425 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422\": container with ID starting with a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422 not found: ID does not exist" containerID="a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422" Oct 30 01:23:48.021876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.021466 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422"} err="failed to get container status \"a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422\": rpc error: code = NotFound desc = could not find container \"a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422\": container with ID starting with a8eb232265c450c6a3a9c4c9e87035cf0669140c8e21fc9dd37634dee9797422 not found: ID does not exist" Oct 30 01:23:48.021876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.021484 4905 scope.go:122] "RemoveContainer" containerID="8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22" Oct 30 01:23:48.035637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.035619 4905 scope.go:122] "RemoveContainer" containerID="8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22" Oct 30 01:23:48.035924 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:48.035904 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22\": container with ID starting with 8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22 not found: ID does not exist" containerID="8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22" Oct 30 01:23:48.036012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.035965 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22"} err="failed to get container status \"8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22\": rpc error: code = NotFound desc = could not find container \"8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22\": container with ID starting with 8103aad60fd41e922123d38a5075ec47a86014edfc9a4cce970491dabde7fc22 not found: ID does not exist" Oct 30 01:23:48.073430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.073399 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-envoy\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.073720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.073701 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-certs\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.074128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.074109 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-credential-socket\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.074577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.074561 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-envoy\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.074752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.074736 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0244d952-9b69-4cdd-bd71-349849e060a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0244d952-9b69-4cdd-bd71-349849e060a5-istio-podinfo\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.074920 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.074908 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-istio-token\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-istio-token\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.075568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.074029 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-certs" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.075686 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.074495 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-credential-socket" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.075771 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.075706 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-socket\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.075988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.075934 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-socket" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.075988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.075973 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-data\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.076189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076013 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-kube-api-access-6rd9j\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-kube-api-access-6rd9j\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.076189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076040 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-kube-api-access-rqz5x\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-kube-api-access-rqz5x\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.076189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076116 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afffce59-7742-4349-abc3-0ab7e9a846a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afffce59-7742-4349-abc3-0ab7e9a846a1-istiod-ca-cert\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.076189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076157 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-certs\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.076434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076245 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-data\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.076434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076276 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-token\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-token\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.076434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076306 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-credential-socket\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.076434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076387 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0244d952-9b69-4cdd-bd71-349849e060a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0244d952-9b69-4cdd-bd71-349849e060a5-istiod-ca-cert\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076471 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-socket\") pod \"0244d952-9b69-4cdd-bd71-349849e060a5\" (UID: \"0244d952-9b69-4cdd-bd71-349849e060a5\") " Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076587 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-podinfo\") pod \"afffce59-7742-4349-abc3-0ab7e9a846a1\" (UID: \"afffce59-7742-4349-abc3-0ab7e9a846a1\") " Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076801 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-certs" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.076985 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-data" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077469 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077489 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077502 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077537 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077552 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077564 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077578 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077592 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077606 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077620 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077633 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxndg\" (UniqueName: \"kubernetes.io/projected/9a911e25-7c2d-424c-8a9e-462891ac0508-kube-api-access-kxndg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077648 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077662 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a911e25-7c2d-424c-8a9e-462891ac0508-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.077674 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a911e25-7c2d-424c-8a9e-462891ac0508-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.078157 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-data" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.078165 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-credential-socket" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.078483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.078314 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afffce59-7742-4349-abc3-0ab7e9a846a1-istiod-ca-cert" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:48.079587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.078896 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0244d952-9b69-4cdd-bd71-349849e060a5-istiod-ca-cert" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:48.079587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.079133 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-socket" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.081209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.081169 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-token" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:48.081628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.081593 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-kube-api-access-6rd9j" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "kube-api-access-6rd9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:48.082760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.082703 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-envoy" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.083477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.083444 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-kube-api-access-rqz5x" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "kube-api-access-rqz5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:48.084803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.084694 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-podinfo" pod "afffce59-7742-4349-abc3-0ab7e9a846a1" (UID: "afffce59-7742-4349-abc3-0ab7e9a846a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:48.085537 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.085492 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-envoy" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:48.085709 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.085670 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-istio-token" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:48.085885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.085863 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0244d952-9b69-4cdd-bd71-349849e060a5-istio-podinfo" pod "0244d952-9b69-4cdd-bd71-349849e060a5" (UID: "0244d952-9b69-4cdd-bd71-349849e060a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:48.179675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.179638 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.179917 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.179900 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180012 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180117 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0244d952-9b69-4cdd-bd71-349849e060a5-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180216 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180300 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180385 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afffce59-7742-4349-abc3-0ab7e9a846a1-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180472 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0244d952-9b69-4cdd-bd71-349849e060a5-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180703 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0244d952-9b69-4cdd-bd71-349849e060a5-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180822 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.180932 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.180920 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rd9j\" (UniqueName: \"kubernetes.io/projected/afffce59-7742-4349-abc3-0ab7e9a846a1-kube-api-access-6rd9j\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.181019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.181007 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqz5x\" (UniqueName: \"kubernetes.io/projected/0244d952-9b69-4cdd-bd71-349849e060a5-kube-api-access-rqz5x\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.181112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.181101 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afffce59-7742-4349-abc3-0ab7e9a846a1-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:48.337861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.337783 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz"] Oct 30 01:23:48.363434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.363398 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h27lz"] Oct 30 01:23:48.399967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.399932 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh"] Oct 30 01:23:48.403806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.403785 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-xhrrh"] Oct 30 01:23:48.439496 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.439468 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw"] Oct 30 01:23:48.443560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:48.443534 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-vrfpw"] Oct 30 01:23:49.036193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:49.036151 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" podUID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:49.192663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:49.192628 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" path="/var/lib/kubelet/pods/0244d952-9b69-4cdd-bd71-349849e060a5/volumes" Oct 30 01:23:49.193250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:49.193233 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" path="/var/lib/kubelet/pods/9a911e25-7c2d-424c-8a9e-462891ac0508/volumes" Oct 30 01:23:49.193716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:49.193703 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" path="/var/lib/kubelet/pods/afffce59-7742-4349-abc3-0ab7e9a846a1/volumes" Oct 30 01:23:50.649553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:50.649490 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:23:50.650059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:50.649620 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:23:51.024160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.024128 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s"] Oct 30 01:23:51.024368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.024325 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" podUID="628e0a8b-2a09-4a0e-a8b1-580b19d88237" containerName="payload-processing" containerID="cri-o://633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a" gracePeriod=30 Oct 30 01:23:51.042146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.042114 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz"] Oct 30 01:23:51.042361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.042335 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" podUID="44210db3-5bda-495e-bdc6-70bc14176599" containerName="payload-pre-processing" containerID="cri-o://1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58" gracePeriod=30 Oct 30 01:23:51.625089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.621808 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:23:51.703557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.703500 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:23:51.789395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.783126 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628e0a8b-2a09-4a0e-a8b1-580b19d88237-kube-api-access-jgn78\" (UniqueName: \"kubernetes.io/projected/628e0a8b-2a09-4a0e-a8b1-580b19d88237-kube-api-access-jgn78\") pod \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " Oct 30 01:23:51.789395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.783213 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44210db3-5bda-495e-bdc6-70bc14176599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44210db3-5bda-495e-bdc6-70bc14176599-plugins-config-volume\") pod \"44210db3-5bda-495e-bdc6-70bc14176599\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " Oct 30 01:23:51.789395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.783473 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628e0a8b-2a09-4a0e-a8b1-580b19d88237-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628e0a8b-2a09-4a0e-a8b1-580b19d88237-plugins-config-volume\") pod \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\" (UID: \"628e0a8b-2a09-4a0e-a8b1-580b19d88237\") " Oct 30 01:23:51.789395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.783574 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44210db3-5bda-495e-bdc6-70bc14176599-kube-api-access-s6rcz\" (UniqueName: \"kubernetes.io/projected/44210db3-5bda-495e-bdc6-70bc14176599-kube-api-access-s6rcz\") pod \"44210db3-5bda-495e-bdc6-70bc14176599\" (UID: \"44210db3-5bda-495e-bdc6-70bc14176599\") " Oct 30 01:23:51.791540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.791486 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628e0a8b-2a09-4a0e-a8b1-580b19d88237-kube-api-access-jgn78" pod "628e0a8b-2a09-4a0e-a8b1-580b19d88237" (UID: "628e0a8b-2a09-4a0e-a8b1-580b19d88237"). InnerVolumeSpecName "kube-api-access-jgn78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:51.792053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.792029 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44210db3-5bda-495e-bdc6-70bc14176599-plugins-config-volume" pod "44210db3-5bda-495e-bdc6-70bc14176599" (UID: "44210db3-5bda-495e-bdc6-70bc14176599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:51.792594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.792575 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628e0a8b-2a09-4a0e-a8b1-580b19d88237-plugins-config-volume" pod "628e0a8b-2a09-4a0e-a8b1-580b19d88237" (UID: "628e0a8b-2a09-4a0e-a8b1-580b19d88237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:51.793374 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.793346 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44210db3-5bda-495e-bdc6-70bc14176599-kube-api-access-s6rcz" pod "44210db3-5bda-495e-bdc6-70bc14176599" (UID: "44210db3-5bda-495e-bdc6-70bc14176599"). InnerVolumeSpecName "kube-api-access-s6rcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:51.890099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.890058 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628e0a8b-2a09-4a0e-a8b1-580b19d88237-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:51.890386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.890353 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rcz\" (UniqueName: \"kubernetes.io/projected/44210db3-5bda-495e-bdc6-70bc14176599-kube-api-access-s6rcz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:51.890502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.890490 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgn78\" (UniqueName: \"kubernetes.io/projected/628e0a8b-2a09-4a0e-a8b1-580b19d88237-kube-api-access-jgn78\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:51.890626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:51.890613 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44210db3-5bda-495e-bdc6-70bc14176599-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:52.001900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.001865 4905 generic.go:356] "Generic (PLEG): container finished" podID="628e0a8b-2a09-4a0e-a8b1-580b19d88237" containerID="633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a" exitCode=0 Oct 30 01:23:52.002149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.001926 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" Oct 30 01:23:52.002149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.001945 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" event={"ID":"628e0a8b-2a09-4a0e-a8b1-580b19d88237","Type":"ContainerDied","Data":"633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a"} Oct 30 01:23:52.002149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.002004 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s" event={"ID":"628e0a8b-2a09-4a0e-a8b1-580b19d88237","Type":"ContainerDied","Data":"94971dba57d5cff31717d784cb650ddf1489bdc30126be36c7d5a56dbbc8b335"} Oct 30 01:23:52.002149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.002028 4905 scope.go:122] "RemoveContainer" containerID="633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a" Oct 30 01:23:52.004811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.004786 4905 generic.go:356] "Generic (PLEG): container finished" podID="44210db3-5bda-495e-bdc6-70bc14176599" containerID="1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58" exitCode=0 Oct 30 01:23:52.004956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.004820 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" event={"ID":"44210db3-5bda-495e-bdc6-70bc14176599","Type":"ContainerDied","Data":"1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58"} Oct 30 01:23:52.004956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.004876 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" event={"ID":"44210db3-5bda-495e-bdc6-70bc14176599","Type":"ContainerDied","Data":"09043268ed39d1360d10ad6304495e781dd4755075dabf16a763bbc7fe15e94c"} Oct 30 01:23:52.004956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.004830 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz" Oct 30 01:23:52.014634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.014487 4905 scope.go:122] "RemoveContainer" containerID="633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a" Oct 30 01:23:52.014810 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:52.014792 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a\": container with ID starting with 633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a not found: ID does not exist" containerID="633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a" Oct 30 01:23:52.014861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.014818 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a"} err="failed to get container status \"633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a\": rpc error: code = NotFound desc = could not find container \"633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a\": container with ID starting with 633934bc41161f0643e155df443694a841a976d278b0cf59ba9cea459223d75a not found: ID does not exist" Oct 30 01:23:52.014861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.014835 4905 scope.go:122] "RemoveContainer" containerID="1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58" Oct 30 01:23:52.025965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.025778 4905 scope.go:122] "RemoveContainer" containerID="1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58" Oct 30 01:23:52.026099 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:23:52.026076 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58\": container with ID starting with 1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58 not found: ID does not exist" containerID="1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58" Oct 30 01:23:52.026149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.026112 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58"} err="failed to get container status \"1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58\": rpc error: code = NotFound desc = could not find container \"1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58\": container with ID starting with 1b5f90c1e7b2ea31d94f678ecacf7799abaf143c91fe25b73e268a19c3a55a58 not found: ID does not exist" Oct 30 01:23:52.059872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.059828 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz"] Oct 30 01:23:52.079237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.079203 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c5a66-765f6c568c-v8flz"] Oct 30 01:23:52.099448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.099419 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s"] Oct 30 01:23:52.103200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.103176 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c5a66-6b55799584-dxz4s"] Oct 30 01:23:52.519922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.519888 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4"] Oct 30 01:23:52.520761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520739 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" Oct 30 01:23:52.520761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520757 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520776 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520781 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520803 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520808 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520823 4905 cpu_manager.go:414] "containerMap: removing container" podUID="628e0a8b-2a09-4a0e-a8b1-580b19d88237" containerName="payload-processing" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520827 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628e0a8b-2a09-4a0e-a8b1-580b19d88237" containerName="payload-processing" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520836 4905 cpu_manager.go:414] "containerMap: removing container" podUID="44210db3-5bda-495e-bdc6-70bc14176599" containerName="payload-pre-processing" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520840 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44210db3-5bda-495e-bdc6-70bc14176599" containerName="payload-pre-processing" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520851 4905 cpu_manager.go:414] "containerMap: removing container" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520855 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520879 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" Oct 30 01:23:52.520969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520883 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" Oct 30 01:23:52.521618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520988 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf94bd3-19e1-4e64-85ca-bda6f576079d" containerName="istio-proxy" Oct 30 01:23:52.521618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.520999 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="628e0a8b-2a09-4a0e-a8b1-580b19d88237" containerName="payload-processing" Oct 30 01:23:52.521618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.521005 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="44210db3-5bda-495e-bdc6-70bc14176599" containerName="payload-pre-processing" Oct 30 01:23:52.521618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.521012 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="afffce59-7742-4349-abc3-0ab7e9a846a1" containerName="istio-proxy" Oct 30 01:23:52.521618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.521021 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0244d952-9b69-4cdd-bd71-349849e060a5" containerName="istio-proxy" Oct 30 01:23:52.521618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.521029 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ea456d-9106-466b-b8eb-bea896ccd4b4" containerName="istio-proxy" Oct 30 01:23:52.521618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.521037 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a911e25-7c2d-424c-8a9e-462891ac0508" containerName="istio-proxy" Oct 30 01:23:52.526597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.526574 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.543739 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.543712 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-954bf6-openshift-default-dockercfg-9c7x5\"" Oct 30 01:23:52.545832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.545811 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4"] Oct 30 01:23:52.704120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704089 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-certs\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704136 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkj7g\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-kube-api-access-gkj7g\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704286 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-podinfo\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704327 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-credential-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704354 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-token\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704508 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-envoy\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704575 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-data\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704647 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.704976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.704806 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde15e70-d861-4e0e-b36a-151e0ac677af-istiod-ca-cert\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.805879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.805798 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-podinfo\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.805879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.805851 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-credential-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.805879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.805880 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-token\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.805910 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-envoy\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.805926 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-data\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.805958 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806087 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde15e70-d861-4e0e-b36a-151e0ac677af-istiod-ca-cert\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806238 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-certs\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806269 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-credential-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806320 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkj7g\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-kube-api-access-gkj7g\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806553 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806563 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-certs\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806728 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-data\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.806858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.806829 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde15e70-d861-4e0e-b36a-151e0ac677af-istiod-ca-cert\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.808381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.808358 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-envoy\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.809086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.809066 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-podinfo\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.827189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.827166 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-token\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.827806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.827785 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkj7g\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-kube-api-access-gkj7g\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:52.847120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:52.847096 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:53.167618 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:23:53.167585 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde15e70_d861_4e0e_b36a_151e0ac677af.slice/crio-4ee3786762f890161df152af76707a4cf87c60c9535f11f65544404c96b3c609 WatchSource:0}: Error finding container 4ee3786762f890161df152af76707a4cf87c60c9535f11f65544404c96b3c609: Status 404 returned error can't find the container with id 4ee3786762f890161df152af76707a4cf87c60c9535f11f65544404c96b3c609 Oct 30 01:23:53.169179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.169151 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4"] Oct 30 01:23:53.171114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.171071 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:53.171223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.171159 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:53.171267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.171218 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:53.194636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.194608 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44210db3-5bda-495e-bdc6-70bc14176599" path="/var/lib/kubelet/pods/44210db3-5bda-495e-bdc6-70bc14176599/volumes" Oct 30 01:23:53.195123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.195106 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628e0a8b-2a09-4a0e-a8b1-580b19d88237" path="/var/lib/kubelet/pods/628e0a8b-2a09-4a0e-a8b1-580b19d88237/volumes" Oct 30 01:23:53.222274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.222247 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm"] Oct 30 01:23:53.262403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.262379 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf"] Oct 30 01:23:53.296779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.296751 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb"] Oct 30 01:23:53.302796 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.302767 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.338185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.338139 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0519f6-openshift-default-dockercfg-jn7wj\"" Oct 30 01:23:53.360662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.359660 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb"] Oct 30 01:23:53.422935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.422728 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220ad5d5-3373-4eb3-a4c2-d9f068297389-istiod-ca-cert\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.422935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.422767 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-socket\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.422935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.422793 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-envoy\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.422935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.422817 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-token\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.423183 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.423049 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-podinfo\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.423218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.423193 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2f5\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-kube-api-access-dd2f5\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.423307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.423291 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-certs\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.423386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.423372 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-data\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.423444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.423431 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-credential-socket\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.591318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.591273 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220ad5d5-3373-4eb3-a4c2-d9f068297389-istiod-ca-cert\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.592352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.592328 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-socket\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.592894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.592876 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-envoy\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.593393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.593374 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-token\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.602182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.602162 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-podinfo\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.602465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.602430 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2f5\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-kube-api-access-dd2f5\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.602686 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.602670 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-certs\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.602889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.602875 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-data\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.603015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.603002 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-credential-socket\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.615657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.592264 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220ad5d5-3373-4eb3-a4c2-d9f068297389-istiod-ca-cert\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.615826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.592813 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-socket\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.615930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.606199 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-certs\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.616027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.606746 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-data\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.616119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.606964 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-credential-socket\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.616206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.615029 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-podinfo\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.677620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.675024 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-envoy\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.677620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.675502 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-token\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.677620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.676215 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2f5\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-kube-api-access-dd2f5\") pod \"e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:53.935881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:53.935788 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:54.124220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.123086 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" event={"ID":"dde15e70-d861-4e0e-b36a-151e0ac677af","Type":"ContainerStarted","Data":"25cf9ef9302ee56a56113bf18fd11493e3f14700dff903816468ec5f26d4eb94"} Oct 30 01:23:54.124220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.123174 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" event={"ID":"dde15e70-d861-4e0e-b36a-151e0ac677af","Type":"ContainerStarted","Data":"4ee3786762f890161df152af76707a4cf87c60c9535f11f65544404c96b3c609"} Oct 30 01:23:54.212196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.212130 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podStartSLOduration=2.212107858 podStartE2EDuration="2.212107858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:54.190077304 +0000 UTC m=+2719.133250497" watchObservedRunningTime="2026-10-30 01:23:54.212107858 +0000 UTC m=+2719.155281052" Oct 30 01:23:54.598554 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:23:54.598509 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220ad5d5_3373_4eb3_a4c2_d9f068297389.slice/crio-3ca2b5a3377a5615e81d1c8d15f42a4194b7ffa182f665a6e4653c47abf206e5 WatchSource:0}: Error finding container 3ca2b5a3377a5615e81d1c8d15f42a4194b7ffa182f665a6e4653c47abf206e5: Status 404 returned error can't find the container with id 3ca2b5a3377a5615e81d1c8d15f42a4194b7ffa182f665a6e4653c47abf206e5 Oct 30 01:23:54.601333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.601288 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb"] Oct 30 01:23:54.604980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.604259 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:54.604980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.604395 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:54.604980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.604503 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:23:54.850692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.850363 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:23:54.850692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.850498 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:23:54.850692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:54.850560 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:23:55.173935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:55.173851 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" event={"ID":"220ad5d5-3373-4eb3-a4c2-d9f068297389","Type":"ContainerStarted","Data":"f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd"} Oct 30 01:23:55.173935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:55.173910 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" event={"ID":"220ad5d5-3373-4eb3-a4c2-d9f068297389","Type":"ContainerStarted","Data":"3ca2b5a3377a5615e81d1c8d15f42a4194b7ffa182f665a6e4653c47abf206e5"} Oct 30 01:23:55.221666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:55.221613 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podStartSLOduration=2.221598312 podStartE2EDuration="2.221598312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:55.217049149 +0000 UTC m=+2720.160222345" watchObservedRunningTime="2026-10-30 01:23:55.221598312 +0000 UTC m=+2720.164771505" Oct 30 01:23:55.849545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:55.847901 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:23:55.849545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:55.847998 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:23:55.950015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:55.949984 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:56.031183 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:56.031146 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:56.323609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:56.299831 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:56.323609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:56.304740 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:23:56.859558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:56.858800 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:23:56.859558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:56.858867 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:23:57.855612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:57.855420 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:23:57.855612 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:57.855476 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:23:58.223221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.223138 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" podUID="6226c647-d9fb-45a4-8364-4ab4550601e6" containerName="payload-processing" containerID="cri-o://07da7c68a55207de877e140534ae0516148fb9fcb4ddeeede66ec66bd69b5eda" gracePeriod=25 Oct 30 01:23:58.270557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.269601 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" podUID="bd71e90d-0219-4f08-8b91-15302757b1c1" containerName="payload-pre-processing" containerID="cri-o://51c1007b0a9ab2f40874c236a14cacdf691ab8be86df76bb5fb93cb71c84fffc" gracePeriod=25 Oct 30 01:23:58.383135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.383100 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm"] Oct 30 01:23:58.384628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.384591 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" containerID="cri-o://00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907" gracePeriod=30 Oct 30 01:23:58.717546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.707677 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224"] Oct 30 01:23:58.717546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.707941 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" podUID="46decaca-579e-43a0-a628-4d41627365ed" containerName="maas-api" containerID="cri-o://087bd7d21efbf1cc474a8bcb129e6963d5b7de8d200b8e92522364c5b1f61d63" gracePeriod=30 Oct 30 01:23:58.852297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.851871 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:23:58.852297 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:58.851930 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:23:59.313580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.304975 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" podUID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:59.613446 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.613411 4905 generic.go:356] "Generic (PLEG): container finished" podID="46decaca-579e-43a0-a628-4d41627365ed" containerID="087bd7d21efbf1cc474a8bcb129e6963d5b7de8d200b8e92522364c5b1f61d63" exitCode=0 Oct 30 01:23:59.613683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.613653 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" event={"ID":"46decaca-579e-43a0-a628-4d41627365ed","Type":"ContainerDied","Data":"087bd7d21efbf1cc474a8bcb129e6963d5b7de8d200b8e92522364c5b1f61d63"} Oct 30 01:23:59.616107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.616083 4905 generic.go:356] "Generic (PLEG): container finished" podID="bd71e90d-0219-4f08-8b91-15302757b1c1" containerID="51c1007b0a9ab2f40874c236a14cacdf691ab8be86df76bb5fb93cb71c84fffc" exitCode=0 Oct 30 01:23:59.616218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.616165 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" event={"ID":"bd71e90d-0219-4f08-8b91-15302757b1c1","Type":"ContainerDied","Data":"51c1007b0a9ab2f40874c236a14cacdf691ab8be86df76bb5fb93cb71c84fffc"} Oct 30 01:23:59.617199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.617171 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 01:23:59.618230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.618213 4905 generic.go:356] "Generic (PLEG): container finished" podID="6226c647-d9fb-45a4-8364-4ab4550601e6" containerID="07da7c68a55207de877e140534ae0516148fb9fcb4ddeeede66ec66bd69b5eda" exitCode=0 Oct 30 01:23:59.618446 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.618327 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" event={"ID":"6226c647-d9fb-45a4-8364-4ab4550601e6","Type":"ContainerDied","Data":"07da7c68a55207de877e140534ae0516148fb9fcb4ddeeede66ec66bd69b5eda"} Oct 30 01:23:59.618617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.618601 4905 scope.go:122] "RemoveContainer" containerID="07da7c68a55207de877e140534ae0516148fb9fcb4ddeeede66ec66bd69b5eda" Oct 30 01:23:59.796856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.795445 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6226c647-d9fb-45a4-8364-4ab4550601e6-kube-api-access-zlx95\" (UniqueName: \"kubernetes.io/projected/6226c647-d9fb-45a4-8364-4ab4550601e6-kube-api-access-zlx95\") pod \"6226c647-d9fb-45a4-8364-4ab4550601e6\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " Oct 30 01:23:59.796856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.795606 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6226c647-d9fb-45a4-8364-4ab4550601e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6226c647-d9fb-45a4-8364-4ab4550601e6-plugins-config-volume\") pod \"6226c647-d9fb-45a4-8364-4ab4550601e6\" (UID: \"6226c647-d9fb-45a4-8364-4ab4550601e6\") " Oct 30 01:23:59.796856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.796799 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6226c647-d9fb-45a4-8364-4ab4550601e6-plugins-config-volume" pod "6226c647-d9fb-45a4-8364-4ab4550601e6" (UID: "6226c647-d9fb-45a4-8364-4ab4550601e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:59.810300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.809907 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 01:23:59.812369 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.812337 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6226c647-d9fb-45a4-8364-4ab4550601e6-kube-api-access-zlx95" pod "6226c647-d9fb-45a4-8364-4ab4550601e6" (UID: "6226c647-d9fb-45a4-8364-4ab4550601e6"). InnerVolumeSpecName "kube-api-access-zlx95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:59.813600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.813495 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:23:59.861541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.847688 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:23:59.861541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.847744 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.909674 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls\") pod \"46decaca-579e-43a0-a628-4d41627365ed\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.909796 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd71e90d-0219-4f08-8b91-15302757b1c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd71e90d-0219-4f08-8b91-15302757b1c1-plugins-config-volume\") pod \"bd71e90d-0219-4f08-8b91-15302757b1c1\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.909840 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-metrics-certs\") pod \"46decaca-579e-43a0-a628-4d41627365ed\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.909875 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46decaca-579e-43a0-a628-4d41627365ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46decaca-579e-43a0-a628-4d41627365ed-openshift-service-ca\") pod \"46decaca-579e-43a0-a628-4d41627365ed\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.909990 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd71e90d-0219-4f08-8b91-15302757b1c1-kube-api-access-hsz2h\" (UniqueName: \"kubernetes.io/projected/bd71e90d-0219-4f08-8b91-15302757b1c1-kube-api-access-hsz2h\") pod \"bd71e90d-0219-4f08-8b91-15302757b1c1\" (UID: \"bd71e90d-0219-4f08-8b91-15302757b1c1\") " Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.910072 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46decaca-579e-43a0-a628-4d41627365ed-kube-api-access-grp26\" (UniqueName: \"kubernetes.io/projected/46decaca-579e-43a0-a628-4d41627365ed-kube-api-access-grp26\") pod \"46decaca-579e-43a0-a628-4d41627365ed\" (UID: \"46decaca-579e-43a0-a628-4d41627365ed\") " Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.911169 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlx95\" (UniqueName: \"kubernetes.io/projected/6226c647-d9fb-45a4-8364-4ab4550601e6-kube-api-access-zlx95\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:59.912545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.911199 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6226c647-d9fb-45a4-8364-4ab4550601e6-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:23:59.913120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.912853 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46decaca-579e-43a0-a628-4d41627365ed-openshift-service-ca" pod "46decaca-579e-43a0-a628-4d41627365ed" (UID: "46decaca-579e-43a0-a628-4d41627365ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:59.921395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:23:59.921356 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd71e90d-0219-4f08-8b91-15302757b1c1-plugins-config-volume" pod "bd71e90d-0219-4f08-8b91-15302757b1c1" (UID: "bd71e90d-0219-4f08-8b91-15302757b1c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:00.014614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.013792 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46decaca-579e-43a0-a628-4d41627365ed-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.014614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.013828 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd71e90d-0219-4f08-8b91-15302757b1c1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.064326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.064236 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls" pod "46decaca-579e-43a0-a628-4d41627365ed" (UID: "46decaca-579e-43a0-a628-4d41627365ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:00.064764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.064735 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-metrics-certs" pod "46decaca-579e-43a0-a628-4d41627365ed" (UID: "46decaca-579e-43a0-a628-4d41627365ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:00.065849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.065820 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46decaca-579e-43a0-a628-4d41627365ed-kube-api-access-grp26" pod "46decaca-579e-43a0-a628-4d41627365ed" (UID: "46decaca-579e-43a0-a628-4d41627365ed"). InnerVolumeSpecName "kube-api-access-grp26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:00.085377 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.084324 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd71e90d-0219-4f08-8b91-15302757b1c1-kube-api-access-hsz2h" pod "bd71e90d-0219-4f08-8b91-15302757b1c1" (UID: "bd71e90d-0219-4f08-8b91-15302757b1c1"). InnerVolumeSpecName "kube-api-access-hsz2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:00.115434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.115401 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsz2h\" (UniqueName: \"kubernetes.io/projected/bd71e90d-0219-4f08-8b91-15302757b1c1-kube-api-access-hsz2h\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.115625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.115436 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grp26\" (UniqueName: \"kubernetes.io/projected/46decaca-579e-43a0-a628-4d41627365ed-kube-api-access-grp26\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.115625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.115462 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.115625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.115475 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46decaca-579e-43a0-a628-4d41627365ed-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.820218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.820112 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" event={"ID":"46decaca-579e-43a0-a628-4d41627365ed","Type":"ContainerDied","Data":"83b1ffd6c970c6671b043c8147cc1165ad5762b73510513bc42b3f1ab2b2de67"} Oct 30 01:24:00.820777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.820194 4905 scope.go:122] "RemoveContainer" containerID="087bd7d21efbf1cc474a8bcb129e6963d5b7de8d200b8e92522364c5b1f61d63" Oct 30 01:24:00.820777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.820481 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224" Oct 30 01:24:00.878564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.871722 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:24:00.878564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.871797 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:24:00.878564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.872501 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" event={"ID":"bd71e90d-0219-4f08-8b91-15302757b1c1","Type":"ContainerDied","Data":"9c9d65fd8f4606d5d67662697cba83750fd5e18fa08e37d033c1a89daefe8595"} Oct 30 01:24:00.883143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.882672 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" event={"ID":"6226c647-d9fb-45a4-8364-4ab4550601e6","Type":"ContainerDied","Data":"cbb91432e1489123a96942556bed1351dd2c2017c4e4645507c240f980ba001a"} Oct 30 01:24:00.883466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.883388 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p"] Oct 30 01:24:00.884163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.884141 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf" Oct 30 01:24:00.888301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.884763 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm" Oct 30 01:24:00.889976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.885930 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6226c647-d9fb-45a4-8364-4ab4550601e6" containerName="payload-processing" Oct 30 01:24:00.890153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.890135 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6226c647-d9fb-45a4-8364-4ab4550601e6" containerName="payload-processing" Oct 30 01:24:00.890337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.890303 4905 cpu_manager.go:414] "containerMap: removing container" podUID="bd71e90d-0219-4f08-8b91-15302757b1c1" containerName="payload-pre-processing" Oct 30 01:24:00.896818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.890503 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd71e90d-0219-4f08-8b91-15302757b1c1" containerName="payload-pre-processing" Oct 30 01:24:00.897010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.896992 4905 cpu_manager.go:414] "containerMap: removing container" podUID="46decaca-579e-43a0-a628-4d41627365ed" containerName="maas-api" Oct 30 01:24:00.897156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.897142 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46decaca-579e-43a0-a628-4d41627365ed" containerName="maas-api" Oct 30 01:24:00.897507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.897489 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6226c647-d9fb-45a4-8364-4ab4550601e6" containerName="payload-processing" Oct 30 01:24:00.897650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.897636 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd71e90d-0219-4f08-8b91-15302757b1c1" containerName="payload-pre-processing" Oct 30 01:24:00.897739 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.897730 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="46decaca-579e-43a0-a628-4d41627365ed" containerName="maas-api" Oct 30 01:24:00.902973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.902956 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d"] Oct 30 01:24:00.906478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.906440 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:00.911150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.911128 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p"] Oct 30 01:24:00.911434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.911374 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:00.922456 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.914928 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d"] Oct 30 01:24:00.931639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.930439 4905 scope.go:122] "RemoveContainer" containerID="51c1007b0a9ab2f40874c236a14cacdf691ab8be86df76bb5fb93cb71c84fffc" Oct 30 01:24:00.944619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.933889 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0519f6-dockercfg-nh2kl\"" Oct 30 01:24:00.944619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.933983 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0519f6\"" Oct 30 01:24:00.986552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.952431 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224"] Oct 30 01:24:00.986552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.954771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8752\" (UniqueName: \"kubernetes.io/projected/70df62fa-e007-4674-b5bc-07d10d2cf858-kube-api-access-j8752\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:00.986552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.954939 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70df62fa-e007-4674-b5bc-07d10d2cf858-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:00.986552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:00.983676 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24d5ce-5579cc8b6b-dd224"] Oct 30 01:24:01.074438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.074336 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndlh\" (UniqueName: \"kubernetes.io/projected/060b152e-dbf2-491a-8d54-f3641368c588-kube-api-access-vndlh\") pod \"payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:01.077170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.074758 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70df62fa-e007-4674-b5bc-07d10d2cf858-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:01.115541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.077597 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060b152e-dbf2-491a-8d54-f3641368c588-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:01.115939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.115905 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8752\" (UniqueName: \"kubernetes.io/projected/70df62fa-e007-4674-b5bc-07d10d2cf858-kube-api-access-j8752\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:01.116548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.090429 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70df62fa-e007-4674-b5bc-07d10d2cf858-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:01.323548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.323063 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060b152e-dbf2-491a-8d54-f3641368c588-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:01.323548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.323175 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndlh\" (UniqueName: \"kubernetes.io/projected/060b152e-dbf2-491a-8d54-f3641368c588-kube-api-access-vndlh\") pod \"payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:01.324561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.324468 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060b152e-dbf2-491a-8d54-f3641368c588-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:01.490542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.453884 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46decaca-579e-43a0-a628-4d41627365ed" path="/var/lib/kubelet/pods/46decaca-579e-43a0-a628-4d41627365ed/volumes" Oct 30 01:24:01.490542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.483137 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndlh\" (UniqueName: \"kubernetes.io/projected/060b152e-dbf2-491a-8d54-f3641368c588-kube-api-access-vndlh\") pod \"payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:01.504545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.502130 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8752\" (UniqueName: \"kubernetes.io/projected/70df62fa-e007-4674-b5bc-07d10d2cf858-kube-api-access-j8752\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:01.542653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.542122 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm"] Oct 30 01:24:01.557090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.548674 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca324d-76fb4f9544-jtzkm"] Oct 30 01:24:01.601245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.601158 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:01.712505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.693329 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:01.879205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.879171 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 01:24:01.879691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.879662 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 01:24:01.880158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.880137 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf"] Oct 30 01:24:01.935556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:01.935132 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca324d-76f785b75f-tb4zf"] Oct 30 01:24:02.047544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.047491 4905 generic.go:356] "Generic (PLEG): container finished" podID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" containerID="6d46f348e1fa0758b37f95b5909feea7a5473105dca23945f26d863bd8b06fd1" exitCode=0 Oct 30 01:24:02.047839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.047816 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" event={"ID":"3399f18d-1f37-4f43-8b91-0bfff4b22e1d","Type":"ContainerDied","Data":"6d46f348e1fa0758b37f95b5909feea7a5473105dca23945f26d863bd8b06fd1"} Oct 30 01:24:02.453014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.452984 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p"] Oct 30 01:24:02.456371 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:24:02.456308 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70df62fa_e007_4674_b5bc_07d10d2cf858.slice/crio-956981e272ceb58922485feb6b3365aa8fa35688f459c59ce4d1de6488bcd12e WatchSource:0}: Error finding container 956981e272ceb58922485feb6b3365aa8fa35688f459c59ce4d1de6488bcd12e: Status 404 returned error can't find the container with id 956981e272ceb58922485feb6b3365aa8fa35688f459c59ce4d1de6488bcd12e Oct 30 01:24:02.538554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.493861 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d"] Oct 30 01:24:02.595392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.595363 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:24:02.748391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.742699 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-kube-api-access-5mxlt\" (UniqueName: \"kubernetes.io/projected/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-kube-api-access-5mxlt\") pod \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " Oct 30 01:24:02.748391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.742840 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-plugins-config-volume\") pod \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\" (UID: \"3399f18d-1f37-4f43-8b91-0bfff4b22e1d\") " Oct 30 01:24:02.748391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.743919 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-plugins-config-volume" pod "3399f18d-1f37-4f43-8b91-0bfff4b22e1d" (UID: "3399f18d-1f37-4f43-8b91-0bfff4b22e1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:02.755902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.755869 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-kube-api-access-5mxlt" pod "3399f18d-1f37-4f43-8b91-0bfff4b22e1d" (UID: "3399f18d-1f37-4f43-8b91-0bfff4b22e1d"). InnerVolumeSpecName "kube-api-access-5mxlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:02.850132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.849399 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mxlt\" (UniqueName: \"kubernetes.io/projected/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-kube-api-access-5mxlt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:02.850132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:02.849436 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3399f18d-1f37-4f43-8b91-0bfff4b22e1d-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:03.058444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.058400 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" event={"ID":"060b152e-dbf2-491a-8d54-f3641368c588","Type":"ContainerStarted","Data":"031e9201535eea0b64351acb7516a35d39a837b6daccf681c2f11257e7e60870"} Oct 30 01:24:03.059090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.058473 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" event={"ID":"060b152e-dbf2-491a-8d54-f3641368c588","Type":"ContainerStarted","Data":"9ce977dd3b4db181a81c571be0ed9534e6aa22ee02d20bfaa0ff34d07eabe192"} Oct 30 01:24:03.059090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.058725 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:03.060867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.060836 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" event={"ID":"70df62fa-e007-4674-b5bc-07d10d2cf858","Type":"ContainerStarted","Data":"74b66c44f5fdd7482583c1d0ac32cd427298a10abc61421a59ebbe3bb5c454d7"} Oct 30 01:24:03.061056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.060901 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" event={"ID":"70df62fa-e007-4674-b5bc-07d10d2cf858","Type":"ContainerStarted","Data":"956981e272ceb58922485feb6b3365aa8fa35688f459c59ce4d1de6488bcd12e"} Oct 30 01:24:03.061144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.061125 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:03.062687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.062656 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" Oct 30 01:24:03.062687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.062664 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496" event={"ID":"3399f18d-1f37-4f43-8b91-0bfff4b22e1d","Type":"ContainerDied","Data":"20df2e91908978139844fbc53b800da98f23bc210ead8b1d88a799ad4f0f1194"} Oct 30 01:24:03.062846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.062717 4905 scope.go:122] "RemoveContainer" containerID="6d46f348e1fa0758b37f95b5909feea7a5473105dca23945f26d863bd8b06fd1" Oct 30 01:24:03.105037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.104889 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" podStartSLOduration=3.104867494 podStartE2EDuration="3.104867494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:03.099832559 +0000 UTC m=+2728.043005754" watchObservedRunningTime="2026-10-30 01:24:03.104867494 +0000 UTC m=+2728.048040688" Oct 30 01:24:03.142469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.142061 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496"] Oct 30 01:24:03.147829 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.147691 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24d5ce-78ccfbb69f-l9496"] Oct 30 01:24:03.186565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.186469 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" podStartSLOduration=3.186451504 podStartE2EDuration="3.186451504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:03.183122729 +0000 UTC m=+2728.126295923" watchObservedRunningTime="2026-10-30 01:24:03.186451504 +0000 UTC m=+2728.129624698" Oct 30 01:24:03.199751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.199721 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" path="/var/lib/kubelet/pods/3399f18d-1f37-4f43-8b91-0bfff4b22e1d/volumes" Oct 30 01:24:03.200268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.200245 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6226c647-d9fb-45a4-8364-4ab4550601e6" path="/var/lib/kubelet/pods/6226c647-d9fb-45a4-8364-4ab4550601e6/volumes" Oct 30 01:24:03.200770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.200750 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd71e90d-0219-4f08-8b91-15302757b1c1" path="/var/lib/kubelet/pods/bd71e90d-0219-4f08-8b91-15302757b1c1/volumes" Oct 30 01:24:03.651634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.651594 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:24:03.652365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.652311 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:24:03.654798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.654775 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:24:03.826290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.825456 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r"] Oct 30 01:24:03.826290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.825712 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" containerID="cri-o://de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2" gracePeriod=30 Oct 30 01:24:03.859362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:03.859272 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.114461 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-kube-api-access-fg9mm\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-kube-api-access-fg9mm\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.114671 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-credential-socket\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.114724 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-socket\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.114855 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-podinfo\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.114892 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-certs\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.114985 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-token\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-token\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.115053 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-data\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.115096 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-envoy\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.115127 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaf339a-6d1f-460e-8b17-57b3494d6707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaf339a-6d1f-460e-8b17-57b3494d6707-istiod-ca-cert\") pod \"8aaf339a-6d1f-460e-8b17-57b3494d6707\" (UID: \"8aaf339a-6d1f-460e-8b17-57b3494d6707\") " Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.143441 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-socket" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:04.148594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.148145 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-credential-socket" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:04.157291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.157101 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-data" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:04.157291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.157206 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-certs" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:04.163551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.163426 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-podinfo" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:04.164472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.164256 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-envoy" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:04.164472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.164315 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-token" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:04.166867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.166826 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-kube-api-access-fg9mm" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "kube-api-access-fg9mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:04.182545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.179871 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaf339a-6d1f-460e-8b17-57b3494d6707-istiod-ca-cert" pod "8aaf339a-6d1f-460e-8b17-57b3494d6707" (UID: "8aaf339a-6d1f-460e-8b17-57b3494d6707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:04.184927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.184460 4905 generic.go:356] "Generic (PLEG): container finished" podID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerID="00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907" exitCode=0 Oct 30 01:24:04.187658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.185237 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" event={"ID":"8aaf339a-6d1f-460e-8b17-57b3494d6707","Type":"ContainerDied","Data":"00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907"} Oct 30 01:24:04.187658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.185332 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" event={"ID":"8aaf339a-6d1f-460e-8b17-57b3494d6707","Type":"ContainerDied","Data":"3d90c84f210c2a54ef61322c7b970f1fb9b00fe36ce782171a240807b665190a"} Oct 30 01:24:04.187658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.185378 4905 scope.go:122] "RemoveContainer" containerID="00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907" Oct 30 01:24:04.187658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.185582 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm" Oct 30 01:24:04.222897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.222864 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223074 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223134 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223149 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223162 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223174 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223186 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaf339a-6d1f-460e-8b17-57b3494d6707-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223197 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaf339a-6d1f-460e-8b17-57b3494d6707-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.223292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.223209 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9mm\" (UniqueName: \"kubernetes.io/projected/8aaf339a-6d1f-460e-8b17-57b3494d6707-kube-api-access-fg9mm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:04.234703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.234683 4905 scope.go:122] "RemoveContainer" containerID="179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141" Oct 30 01:24:04.250254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.250231 4905 scope.go:122] "RemoveContainer" containerID="00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907" Oct 30 01:24:04.250660 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:04.250628 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907\": container with ID starting with 00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907 not found: ID does not exist" containerID="00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907" Oct 30 01:24:04.250781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.250670 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907"} err="failed to get container status \"00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907\": rpc error: code = NotFound desc = could not find container \"00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907\": container with ID starting with 00e160570e34de37f6df3b935fd100ea6ea5d5cf0b52fbe845a16ea69a361907 not found: ID does not exist" Oct 30 01:24:04.250781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.250695 4905 scope.go:122] "RemoveContainer" containerID="179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141" Oct 30 01:24:04.250984 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:04.250964 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141\": container with ID starting with 179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141 not found: ID does not exist" containerID="179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141" Oct 30 01:24:04.251047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.250991 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141"} err="failed to get container status \"179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141\": rpc error: code = NotFound desc = could not find container \"179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141\": container with ID starting with 179f7ba622420a3f586aaaa246e8dcef6a9494b78a3a08c19825d0e3bbc36141 not found: ID does not exist" Oct 30 01:24:04.269177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.269142 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm"] Oct 30 01:24:04.273642 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:04.273614 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaf339a_6d1f_460e_8b17_57b3494d6707.slice/crio-3d90c84f210c2a54ef61322c7b970f1fb9b00fe36ce782171a240807b665190a\": RecentStats: unable to find data in memory cache]" Oct 30 01:24:04.283798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:04.283768 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c5a66-openshift-default-f75c98fdf-6d4cm"] Oct 30 01:24:05.195086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:05.195043 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" path="/var/lib/kubelet/pods/8aaf339a-6d1f-460e-8b17-57b3494d6707/volumes" Oct 30 01:24:05.719173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:05.719140 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:24:05.719387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:05.719193 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:06.222300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:06.222268 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb"] Oct 30 01:24:06.222870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:06.222553 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" containerID="cri-o://67f9a9da8e619e589393cba62b0798fc0869de6d27450f275117d8c169d13068" gracePeriod=30 Oct 30 01:24:09.135947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.135917 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:24:09.159272 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159242 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-certs\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159329 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-credential-socket\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159361 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44d7288-6855-4519-be9e-ab84d52bbb57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d7288-6855-4519-be9e-ab84d52bbb57-istiod-ca-cert\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159550 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-socket\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159556 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-certs" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:09.159597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159584 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-token\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-token\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159655 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-data\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159660 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-credential-socket" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:09.159774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159698 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-podinfo\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159736 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-envoy\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159778 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-socket" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:09.159990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159788 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-kube-api-access-2xxjz\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-kube-api-access-2xxjz\") pod \"b44d7288-6855-4519-be9e-ab84d52bbb57\" (UID: \"b44d7288-6855-4519-be9e-ab84d52bbb57\") " Oct 30 01:24:09.159990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159882 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-data" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:09.159990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.159926 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44d7288-6855-4519-be9e-ab84d52bbb57-istiod-ca-cert" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:09.160846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.160820 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.160846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.160850 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d7288-6855-4519-be9e-ab84d52bbb57-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.161047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.160867 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.161047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.160882 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.161047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.160896 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.162361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.162334 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-kube-api-access-2xxjz" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "kube-api-access-2xxjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:09.162787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.162756 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-token" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:09.162787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.162775 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-podinfo" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:09.162899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.162817 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-envoy" pod "b44d7288-6855-4519-be9e-ab84d52bbb57" (UID: "b44d7288-6855-4519-be9e-ab84d52bbb57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:09.262474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.262425 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.262474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.262462 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.262474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.262477 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d7288-6855-4519-be9e-ab84d52bbb57-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.262815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.262490 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxjz\" (UniqueName: \"kubernetes.io/projected/b44d7288-6855-4519-be9e-ab84d52bbb57-kube-api-access-2xxjz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:09.270716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.270683 4905 generic.go:356] "Generic (PLEG): container finished" podID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerID="de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2" exitCode=0 Oct 30 01:24:09.271150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.271125 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" event={"ID":"b44d7288-6855-4519-be9e-ab84d52bbb57","Type":"ContainerDied","Data":"de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2"} Oct 30 01:24:09.271326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.271311 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" event={"ID":"b44d7288-6855-4519-be9e-ab84d52bbb57","Type":"ContainerDied","Data":"86cf8cbf0773db87bdce52073610682a1dd88a89982e3563074594bb631d9a06"} Oct 30 01:24:09.271441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.271430 4905 scope.go:122] "RemoveContainer" containerID="de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2" Oct 30 01:24:09.271700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.271677 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r" Oct 30 01:24:09.308333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.308311 4905 scope.go:122] "RemoveContainer" containerID="a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605" Oct 30 01:24:09.325436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.325411 4905 scope.go:122] "RemoveContainer" containerID="de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2" Oct 30 01:24:09.325828 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:09.325798 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2\": container with ID starting with de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2 not found: ID does not exist" containerID="de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2" Oct 30 01:24:09.325910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.325841 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2"} err="failed to get container status \"de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2\": rpc error: code = NotFound desc = could not find container \"de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2\": container with ID starting with de47382b0b131fec8f4678d5fc24d0a5924d7edf6549d0d26993008491108ed2 not found: ID does not exist" Oct 30 01:24:09.325910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.325864 4905 scope.go:122] "RemoveContainer" containerID="a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605" Oct 30 01:24:09.326192 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:09.326168 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605\": container with ID starting with a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605 not found: ID does not exist" containerID="a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605" Oct 30 01:24:09.326260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.326204 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605"} err="failed to get container status \"a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605\": rpc error: code = NotFound desc = could not find container \"a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605\": container with ID starting with a7b0b4c4f3f10e26fc132a0d93114b784ec65d8188a36c5ffc5fda06e3e64605 not found: ID does not exist" Oct 30 01:24:09.340426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.340400 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r"] Oct 30 01:24:09.350327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.350292 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-96bd13-openshift-default-74d4644bc-pxg9r"] Oct 30 01:24:09.354014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:09.353992 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5"] Oct 30 01:24:11.122548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.119962 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:24:11.122548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.120121 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:11.195090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.195048 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" path="/var/lib/kubelet/pods/b44d7288-6855-4519-be9e-ab84d52bbb57/volumes" Oct 30 01:24:11.343904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.343873 4905 generic.go:356] "Generic (PLEG): container finished" podID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerID="67f9a9da8e619e589393cba62b0798fc0869de6d27450f275117d8c169d13068" exitCode=0 Oct 30 01:24:11.344030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.343956 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" event={"ID":"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a","Type":"ContainerDied","Data":"67f9a9da8e619e589393cba62b0798fc0869de6d27450f275117d8c169d13068"} Oct 30 01:24:11.344093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.344030 4905 scope.go:122] "RemoveContainer" containerID="04486989f9ddf11a9951f190939bd89dc151ad9433ccecfeb3f313941f4191ef" Oct 30 01:24:11.513596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.513572 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548090 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-token\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-token\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548165 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-credential-socket\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548204 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-socket\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548271 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-data\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548346 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-certs\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548392 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-envoy\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548476 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-kube-api-access-7sz2m\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-kube-api-access-7sz2m\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548564 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-credential-socket" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548756 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-certs" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.549101 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-socket" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.549281 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-data" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.549545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.548513 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istiod-ca-cert\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.550291 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.549615 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-podinfo\") pod \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\" (UID: \"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a\") " Oct 30 01:24:11.550350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.550332 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.550410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.550356 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.550410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.550383 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.550410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.550398 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.566574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.553118 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-token" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:11.566574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.553305 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-podinfo" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:11.566574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.553391 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-kube-api-access-7sz2m" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "kube-api-access-7sz2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:11.566574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.553601 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istiod-ca-cert" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:11.566574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.555790 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-envoy" pod "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" (UID: "7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.652695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.652659 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.652695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.652695 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sz2m\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-kube-api-access-7sz2m\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.652965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.652709 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.652965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.652723 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.652965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:11.652737 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:12.353253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:12.353217 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" event={"ID":"7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a","Type":"ContainerDied","Data":"c3b0165037b2fa8859811a5408601f7fd27782ae3aa71a9edec07cd4e1f27b58"} Oct 30 01:24:12.353811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:12.353284 4905 scope.go:122] "RemoveContainer" containerID="67f9a9da8e619e589393cba62b0798fc0869de6d27450f275117d8c169d13068" Oct 30 01:24:12.353811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:12.353225 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb" Oct 30 01:24:12.416223 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:12.416190 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:24:12.423906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:12.423881 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb"] Oct 30 01:24:12.437472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:12.437445 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-ncvfb"] Oct 30 01:24:13.192286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:13.192255 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" path="/var/lib/kubelet/pods/7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a/volumes" Oct 30 01:24:14.209920 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:14.209891 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:24:14.210501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:14.209988 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:24:14.354469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:14.354422 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" podUID="407de83a-8478-47d5-8fce-4ea9509263bb" containerName="payload-pre-processing" containerID="cri-o://378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a" gracePeriod=25 Oct 30 01:24:17.237166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:17.237077 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv"] Oct 30 01:24:17.261011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:17.260970 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs"] Oct 30 01:24:19.003903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.003862 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc"] Oct 30 01:24:19.023910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.023863 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" podUID="dc1d798f-f790-41d3-b9f0-773cf307b368" containerName="maas-api" containerID="cri-o://301d9d1696ac826256be2feeb65df9b77d71af23d0d24bcf69b487a73211413c" gracePeriod=30 Oct 30 01:24:19.555548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.555432 4905 generic.go:356] "Generic (PLEG): container finished" podID="dc1d798f-f790-41d3-b9f0-773cf307b368" containerID="301d9d1696ac826256be2feeb65df9b77d71af23d0d24bcf69b487a73211413c" exitCode=0 Oct 30 01:24:19.555890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.555864 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" event={"ID":"dc1d798f-f790-41d3-b9f0-773cf307b368","Type":"ContainerDied","Data":"301d9d1696ac826256be2feeb65df9b77d71af23d0d24bcf69b487a73211413c"} Oct 30 01:24:19.687656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.687620 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:24:19.847213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.847005 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1d798f-f790-41d3-b9f0-773cf307b368-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d798f-f790-41d3-b9f0-773cf307b368-openshift-service-ca\") pod \"dc1d798f-f790-41d3-b9f0-773cf307b368\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " Oct 30 01:24:19.847213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.847099 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-metrics-certs\") pod \"dc1d798f-f790-41d3-b9f0-773cf307b368\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " Oct 30 01:24:19.847213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.847203 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1d798f-f790-41d3-b9f0-773cf307b368-kube-api-access-knfqw\" (UniqueName: \"kubernetes.io/projected/dc1d798f-f790-41d3-b9f0-773cf307b368-kube-api-access-knfqw\") pod \"dc1d798f-f790-41d3-b9f0-773cf307b368\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " Oct 30 01:24:19.853561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.848375 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1d798f-f790-41d3-b9f0-773cf307b368-openshift-service-ca" pod "dc1d798f-f790-41d3-b9f0-773cf307b368" (UID: "dc1d798f-f790-41d3-b9f0-773cf307b368"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:19.853561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.849029 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-maas-api-tls\") pod \"dc1d798f-f790-41d3-b9f0-773cf307b368\" (UID: \"dc1d798f-f790-41d3-b9f0-773cf307b368\") " Oct 30 01:24:19.853561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.849772 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d798f-f790-41d3-b9f0-773cf307b368-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.858621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.858587 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1d798f-f790-41d3-b9f0-773cf307b368-kube-api-access-knfqw" pod "dc1d798f-f790-41d3-b9f0-773cf307b368" (UID: "dc1d798f-f790-41d3-b9f0-773cf307b368"). InnerVolumeSpecName "kube-api-access-knfqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:19.865348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.865320 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-maas-api-tls" pod "dc1d798f-f790-41d3-b9f0-773cf307b368" (UID: "dc1d798f-f790-41d3-b9f0-773cf307b368"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:19.865495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.865448 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-metrics-certs" pod "dc1d798f-f790-41d3-b9f0-773cf307b368" (UID: "dc1d798f-f790-41d3-b9f0-773cf307b368"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:19.951343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.951309 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.951343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.951338 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knfqw\" (UniqueName: \"kubernetes.io/projected/dc1d798f-f790-41d3-b9f0-773cf307b368-kube-api-access-knfqw\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.951605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:19.951359 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1d798f-f790-41d3-b9f0-773cf307b368-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:20.260843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:20.260810 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" podUID="407de83a-8478-47d5-8fce-4ea9509263bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:20.565793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:20.565703 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" Oct 30 01:24:20.565793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:20.565730 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc" event={"ID":"dc1d798f-f790-41d3-b9f0-773cf307b368","Type":"ContainerDied","Data":"5d2c5cbe921ad577fb4a43a0d41d64d1181f36f166162c14f992d74079de98b0"} Oct 30 01:24:20.566043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:20.565795 4905 scope.go:122] "RemoveContainer" containerID="301d9d1696ac826256be2feeb65df9b77d71af23d0d24bcf69b487a73211413c" Oct 30 01:24:20.622265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:20.622238 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc"] Oct 30 01:24:20.636112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:20.636082 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-42d56b-7b88495754-d9wxc"] Oct 30 01:24:21.193412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:21.193372 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1d798f-f790-41d3-b9f0-773cf307b368" path="/var/lib/kubelet/pods/dc1d798f-f790-41d3-b9f0-773cf307b368/volumes" Oct 30 01:24:22.237633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:22.237588 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" podUID="b4110fc9-997e-46ed-9968-9b39240e7afb" containerName="payload-processing" containerID="cri-o://a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4" gracePeriod=25 Oct 30 01:24:22.261324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:22.261286 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" podUID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" containerName="payload-pre-processing" containerID="cri-o://779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a" gracePeriod=25 Oct 30 01:24:22.979274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:22.979235 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn"] Oct 30 01:24:22.979616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:22.979580 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" containerID="cri-o://01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51" gracePeriod=30 Oct 30 01:24:25.957563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:25.957506 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk"] Oct 30 01:24:25.958768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:25.958728 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" containerID="cri-o://cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849" gracePeriod=30 Oct 30 01:24:26.921349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:26.921314 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r"] Oct 30 01:24:26.921610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:26.921581 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" containerID="cri-o://7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb" gracePeriod=30 Oct 30 01:24:27.379743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:27.379702 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:24:27.380157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:27.379779 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:28.285559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.285511 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:24:28.383687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.383612 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9370e3cd-a5c6-4b32-833b-68195297a32f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9370e3cd-a5c6-4b32-833b-68195297a32f-istiod-ca-cert\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.383716 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-credential-socket\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.383749 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-token\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-token\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.383781 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-kube-api-access-df9v8\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-kube-api-access-df9v8\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.383864 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-data\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.383906 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-socket\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.383933 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-certs\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384012 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-podinfo\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384032 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-envoy\") pod \"9370e3cd-a5c6-4b32-833b-68195297a32f\" (UID: \"9370e3cd-a5c6-4b32-833b-68195297a32f\") " Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384050 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-credential-socket" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:28.384159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384080 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9370e3cd-a5c6-4b32-833b-68195297a32f-istiod-ca-cert" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:28.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384277 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-data" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:28.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384376 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-certs" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:28.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384617 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384638 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384651 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9370e3cd-a5c6-4b32-833b-68195297a32f-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384664 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.385836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.384887 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-socket" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:28.387161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.386968 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-token" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:28.387622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.387587 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-envoy" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:28.387719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.387706 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-podinfo" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:28.388322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.388301 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-kube-api-access-df9v8" pod "9370e3cd-a5c6-4b32-833b-68195297a32f" (UID: "9370e3cd-a5c6-4b32-833b-68195297a32f"). InnerVolumeSpecName "kube-api-access-df9v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:28.486145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.486115 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.486145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.486141 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.486304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.486153 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.486304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.486161 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9v8\" (UniqueName: \"kubernetes.io/projected/9370e3cd-a5c6-4b32-833b-68195297a32f-kube-api-access-df9v8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.486304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.486170 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9370e3cd-a5c6-4b32-833b-68195297a32f-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:28.707724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.707602 4905 generic.go:356] "Generic (PLEG): container finished" podID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerID="01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51" exitCode=0 Oct 30 01:24:28.707724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.707669 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" Oct 30 01:24:28.707987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.707811 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" event={"ID":"9370e3cd-a5c6-4b32-833b-68195297a32f","Type":"ContainerDied","Data":"01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51"} Oct 30 01:24:28.707987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.707865 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn" event={"ID":"9370e3cd-a5c6-4b32-833b-68195297a32f","Type":"ContainerDied","Data":"d5cf4ebc1b913dc057398c49c162628a13ba5eabf8e8737bb4fd0d891acd44a6"} Oct 30 01:24:28.707987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.707894 4905 scope.go:122] "RemoveContainer" containerID="01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51" Oct 30 01:24:28.720782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.720761 4905 scope.go:122] "RemoveContainer" containerID="9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a" Oct 30 01:24:28.732905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.732884 4905 scope.go:122] "RemoveContainer" containerID="01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51" Oct 30 01:24:28.733208 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:28.733190 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51\": container with ID starting with 01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51 not found: ID does not exist" containerID="01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51" Oct 30 01:24:28.733289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.733216 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51"} err="failed to get container status \"01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51\": rpc error: code = NotFound desc = could not find container \"01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51\": container with ID starting with 01c1675c392902cd5a46a16cdd9c542f2dea2b355d0aa0cfcb909ae2bdb26e51 not found: ID does not exist" Oct 30 01:24:28.733289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.733232 4905 scope.go:122] "RemoveContainer" containerID="9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a" Oct 30 01:24:28.733548 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:28.733508 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a\": container with ID starting with 9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a not found: ID does not exist" containerID="9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a" Oct 30 01:24:28.733624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.733556 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a"} err="failed to get container status \"9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a\": rpc error: code = NotFound desc = could not find container \"9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a\": container with ID starting with 9c52c3dd7acaa8fec61cfd9ea8852c2505a08cea25104852ade3a49723739d9a not found: ID does not exist" Oct 30 01:24:28.768380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.768357 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn"] Oct 30 01:24:28.773830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:28.773809 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-ln4pn"] Oct 30 01:24:29.191956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:29.191919 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" path="/var/lib/kubelet/pods/9370e3cd-a5c6-4b32-833b-68195297a32f/volumes" Oct 30 01:24:29.680545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:29.679626 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" podUID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:29.782543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:29.781754 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" podUID="b4110fc9-997e-46ed-9968-9b39240e7afb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:30.259761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:30.259723 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" podUID="407de83a-8478-47d5-8fce-4ea9509263bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:30.602271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:30.602174 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:24:30.602271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:30.602240 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:31.291562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.291501 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 01:24:31.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.384541 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-token\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-token\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.384618 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-credential-socket\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.384664 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istiod-ca-cert\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.384791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.384742 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-envoy\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.385161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.384865 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-data\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.385161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.384904 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-credential-socket" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.385161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.384950 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-socket\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.385161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.385018 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-certs\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.385161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.385034 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istiod-ca-cert" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:31.385454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.385237 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-socket" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.385454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.385300 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-kube-api-access-vxfvt\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-kube-api-access-vxfvt\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.385454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.385354 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-podinfo\") pod \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\" (UID: \"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290\") " Oct 30 01:24:31.385454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.385395 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-data" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.385454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.385277 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-certs" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.386774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.386110 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.386774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.386136 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.386774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.386149 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.386774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.386163 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.386774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.386177 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.387594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.387568 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-envoy" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.387728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.387703 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-token" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:31.388223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.388179 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-kube-api-access-vxfvt" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "kube-api-access-vxfvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:31.388223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.388185 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-podinfo" pod "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" (UID: "77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:31.488512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.488469 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxfvt\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-kube-api-access-vxfvt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.488727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.488511 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.488727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.488566 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.488727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.488578 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.822969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.822936 4905 generic.go:356] "Generic (PLEG): container finished" podID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerID="cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849" exitCode=0 Oct 30 01:24:31.823174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.822977 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" event={"ID":"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290","Type":"ContainerDied","Data":"cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849"} Oct 30 01:24:31.823174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.822993 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" Oct 30 01:24:31.823174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.823020 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk" event={"ID":"77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290","Type":"ContainerDied","Data":"407dcea5d7bf089aeea44d1b47f534b336688a3155a97de043e9a98a2ccb146e"} Oct 30 01:24:31.823174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.823041 4905 scope.go:122] "RemoveContainer" containerID="cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849" Oct 30 01:24:31.834984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.834960 4905 scope.go:122] "RemoveContainer" containerID="9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7" Oct 30 01:24:31.846131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.846111 4905 scope.go:122] "RemoveContainer" containerID="cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849" Oct 30 01:24:31.846442 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:31.846422 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849\": container with ID starting with cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849 not found: ID does not exist" containerID="cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849" Oct 30 01:24:31.846550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.846452 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849"} err="failed to get container status \"cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849\": rpc error: code = NotFound desc = could not find container \"cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849\": container with ID starting with cfc1282be0a7b12f5461049e9bebebd512933058d8ca69ead5de9eb19023a849 not found: ID does not exist" Oct 30 01:24:31.846550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.846475 4905 scope.go:122] "RemoveContainer" containerID="9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7" Oct 30 01:24:31.846826 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:31.846807 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7\": container with ID starting with 9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7 not found: ID does not exist" containerID="9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7" Oct 30 01:24:31.846866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.846834 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7"} err="failed to get container status \"9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7\": rpc error: code = NotFound desc = could not find container \"9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7\": container with ID starting with 9bce1df5bcebeff6b57f137aeb07613beeffad41cb7361f212693e045fab50b7 not found: ID does not exist" Oct 30 01:24:31.892114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.892083 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk"] Oct 30 01:24:31.905824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:31.905799 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-j2kfk"] Oct 30 01:24:32.230049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.230025 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:24:32.401281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401203 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-kube-api-access-z2sn9\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-kube-api-access-z2sn9\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401270 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-data\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401306 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-certs\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401395 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-envoy\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401429 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-socket\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401573 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-credential-socket\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401624 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istiod-ca-cert\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401650 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-token\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-token\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401669 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-data" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401691 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-podinfo\") pod \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\" (UID: \"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a\") " Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401737 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-socket" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401780 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-certs" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:32.401813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.401797 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-credential-socket" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:32.402593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.402168 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istiod-ca-cert" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:32.402593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.402399 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.402593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.402421 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.402593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.402437 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.402593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.402451 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.402593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.402466 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.403788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.403765 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-kube-api-access-z2sn9" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "kube-api-access-z2sn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:32.403948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.403932 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-token" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:32.403992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.403939 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-envoy" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:32.404132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.404114 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-podinfo" pod "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" (UID: "b1cde45c-ce1c-4b18-81c7-a0edb5afd13a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:32.504175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.504134 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.504175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.504163 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.504175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.504176 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.504175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.504184 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2sn9\" (UniqueName: \"kubernetes.io/projected/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a-kube-api-access-z2sn9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.836695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.836659 4905 generic.go:356] "Generic (PLEG): container finished" podID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerID="7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb" exitCode=0 Oct 30 01:24:32.836909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.836727 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" Oct 30 01:24:32.836909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.836756 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" event={"ID":"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a","Type":"ContainerDied","Data":"7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb"} Oct 30 01:24:32.836909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.836821 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r" event={"ID":"b1cde45c-ce1c-4b18-81c7-a0edb5afd13a","Type":"ContainerDied","Data":"f1e2860122e32802c101f3ef9e93c78ea51e2b3dbd8180685a8e7a04b065c19f"} Oct 30 01:24:32.836909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.836853 4905 scope.go:122] "RemoveContainer" containerID="7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb" Oct 30 01:24:32.850772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.850752 4905 scope.go:122] "RemoveContainer" containerID="1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd" Oct 30 01:24:32.861866 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.861848 4905 scope.go:122] "RemoveContainer" containerID="7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb" Oct 30 01:24:32.862155 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:32.862130 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb\": container with ID starting with 7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb not found: ID does not exist" containerID="7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb" Oct 30 01:24:32.862218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.862167 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb"} err="failed to get container status \"7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb\": rpc error: code = NotFound desc = could not find container \"7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb\": container with ID starting with 7c4b9bdc3fccae65adb4979cb9f9a725913e7a58b0436ca95bd18951eae25dcb not found: ID does not exist" Oct 30 01:24:32.862218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.862194 4905 scope.go:122] "RemoveContainer" containerID="1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd" Oct 30 01:24:32.862485 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:32.862465 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd\": container with ID starting with 1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd not found: ID does not exist" containerID="1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd" Oct 30 01:24:32.862560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.862491 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd"} err="failed to get container status \"1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd\": rpc error: code = NotFound desc = could not find container \"1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd\": container with ID starting with 1a3d4bb059fa8f62553259140babaac31fbc72896abc13c8623a7f3791f1e5bd not found: ID does not exist" Oct 30 01:24:32.900138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.900103 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r"] Oct 30 01:24:32.914043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:32.914010 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-2qn4r"] Oct 30 01:24:33.246557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:33.236494 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" path="/var/lib/kubelet/pods/77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290/volumes" Oct 30 01:24:33.250567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:33.247354 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" path="/var/lib/kubelet/pods/b1cde45c-ce1c-4b18-81c7-a0edb5afd13a/volumes" Oct 30 01:24:34.612416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.612288 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f"] Oct 30 01:24:34.656460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.656429 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq"] Oct 30 01:24:34.716962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.716930 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:24:34.895629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.894167 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407de83a-8478-47d5-8fce-4ea9509263bb-kube-api-access-p8jpf\" (UniqueName: \"kubernetes.io/projected/407de83a-8478-47d5-8fce-4ea9509263bb-kube-api-access-p8jpf\") pod \"407de83a-8478-47d5-8fce-4ea9509263bb\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " Oct 30 01:24:34.895629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.894311 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407de83a-8478-47d5-8fce-4ea9509263bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407de83a-8478-47d5-8fce-4ea9509263bb-plugins-config-volume\") pod \"407de83a-8478-47d5-8fce-4ea9509263bb\" (UID: \"407de83a-8478-47d5-8fce-4ea9509263bb\") " Oct 30 01:24:34.907859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.907797 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407de83a-8478-47d5-8fce-4ea9509263bb-plugins-config-volume" pod "407de83a-8478-47d5-8fce-4ea9509263bb" (UID: "407de83a-8478-47d5-8fce-4ea9509263bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:34.914666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.914636 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407de83a-8478-47d5-8fce-4ea9509263bb-kube-api-access-p8jpf" pod "407de83a-8478-47d5-8fce-4ea9509263bb" (UID: "407de83a-8478-47d5-8fce-4ea9509263bb"). InnerVolumeSpecName "kube-api-access-p8jpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:34.918594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.918564 4905 generic.go:356] "Generic (PLEG): container finished" podID="407de83a-8478-47d5-8fce-4ea9509263bb" containerID="378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a" exitCode=0 Oct 30 01:24:34.918744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.918672 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" Oct 30 01:24:34.918813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.918794 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" event={"ID":"407de83a-8478-47d5-8fce-4ea9509263bb","Type":"ContainerDied","Data":"378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a"} Oct 30 01:24:34.918874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.918841 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5" event={"ID":"407de83a-8478-47d5-8fce-4ea9509263bb","Type":"ContainerDied","Data":"014a3761cf6329df69171033abfce4e3fccd21f565a753e5960e568311e5a7e7"} Oct 30 01:24:34.918874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.918861 4905 scope.go:122] "RemoveContainer" containerID="378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a" Oct 30 01:24:34.931592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.931576 4905 scope.go:122] "RemoveContainer" containerID="378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a" Oct 30 01:24:34.931867 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:34.931851 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a\": container with ID starting with 378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a not found: ID does not exist" containerID="378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a" Oct 30 01:24:34.931912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.931877 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a"} err="failed to get container status \"378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a\": rpc error: code = NotFound desc = could not find container \"378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a\": container with ID starting with 378f319cfa0a50dc7fe00129b8855d52252ecfc8ac5dbaf26414f4ff194c7b4a not found: ID does not exist" Oct 30 01:24:34.976199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.976171 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5"] Oct 30 01:24:34.984160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.984128 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b05110-855c94fc99-hg7z5"] Oct 30 01:24:34.997376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.997344 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407de83a-8478-47d5-8fce-4ea9509263bb-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:34.997552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:34.997380 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8jpf\" (UniqueName: \"kubernetes.io/projected/407de83a-8478-47d5-8fce-4ea9509263bb-kube-api-access-p8jpf\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:35.194013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:35.193938 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407de83a-8478-47d5-8fce-4ea9509263bb" path="/var/lib/kubelet/pods/407de83a-8478-47d5-8fce-4ea9509263bb/volumes" Oct 30 01:24:37.026966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:37.026933 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67"] Oct 30 01:24:37.876071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:37.876039 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk"] Oct 30 01:24:39.613256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:39.613213 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" podUID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" containerName="payload-processing" containerID="cri-o://15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce" gracePeriod=25 Oct 30 01:24:39.650394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:39.650349 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" podUID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:39.650553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:39.650392 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" podUID="b4110fc9-997e-46ed-9968-9b39240e7afb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:39.657453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:39.657424 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" podUID="a469becb-c0d0-40cc-b426-3287026d49aa" containerName="payload-pre-processing" containerID="cri-o://bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a" gracePeriod=25 Oct 30 01:24:40.512387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:40.512348 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" podUID="a469becb-c0d0-40cc-b426-3287026d49aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:41.522073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:41.522031 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" podUID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:42.027301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.027233 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" podUID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" containerName="payload-pre-processing" containerID="cri-o://49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4" gracePeriod=25 Oct 30 01:24:42.422658 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:42.422619 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:24:42.564506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.564475 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:24:42.628367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.628336 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:24:42.636113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.636084 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4110fc9-997e-46ed-9968-9b39240e7afb-kube-api-access-wj4w7\" (UniqueName: \"kubernetes.io/projected/b4110fc9-997e-46ed-9968-9b39240e7afb-kube-api-access-wj4w7\") pod \"b4110fc9-997e-46ed-9968-9b39240e7afb\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " Oct 30 01:24:42.636381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.636362 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4110fc9-997e-46ed-9968-9b39240e7afb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4110fc9-997e-46ed-9968-9b39240e7afb-plugins-config-volume\") pod \"b4110fc9-997e-46ed-9968-9b39240e7afb\" (UID: \"b4110fc9-997e-46ed-9968-9b39240e7afb\") " Oct 30 01:24:42.636897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.636867 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4110fc9-997e-46ed-9968-9b39240e7afb-plugins-config-volume" pod "b4110fc9-997e-46ed-9968-9b39240e7afb" (UID: "b4110fc9-997e-46ed-9968-9b39240e7afb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:42.639668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.639641 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4110fc9-997e-46ed-9968-9b39240e7afb-kube-api-access-wj4w7" pod "b4110fc9-997e-46ed-9968-9b39240e7afb" (UID: "b4110fc9-997e-46ed-9968-9b39240e7afb"). InnerVolumeSpecName "kube-api-access-wj4w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:42.737688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.737656 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-kube-api-access-d4fhq\" (UniqueName: \"kubernetes.io/projected/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-kube-api-access-d4fhq\") pod \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " Oct 30 01:24:42.737865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.737812 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-plugins-config-volume\") pod \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\" (UID: \"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0\") " Oct 30 01:24:42.738201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.738179 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4110fc9-997e-46ed-9968-9b39240e7afb-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:42.738201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.738197 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4w7\" (UniqueName: \"kubernetes.io/projected/b4110fc9-997e-46ed-9968-9b39240e7afb-kube-api-access-wj4w7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:42.738420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.738229 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-plugins-config-volume" pod "593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" (UID: "593b152c-46a1-4f22-b9b0-bc47a7ad2aa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:42.740019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.739997 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-kube-api-access-d4fhq" pod "593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" (UID: "593b152c-46a1-4f22-b9b0-bc47a7ad2aa0"). InnerVolumeSpecName "kube-api-access-d4fhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:42.839413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.839321 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:42.839413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.839353 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4fhq\" (UniqueName: \"kubernetes.io/projected/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0-kube-api-access-d4fhq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:42.876838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:42.876797 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" podUID="4931e93a-0f20-47c1-9f73-ffe763e183b1" containerName="payload-pre-processing" containerID="cri-o://e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458" gracePeriod=25 Oct 30 01:24:43.012824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.012787 4905 generic.go:356] "Generic (PLEG): container finished" podID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" containerID="779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a" exitCode=0 Oct 30 01:24:43.012973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.012839 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" Oct 30 01:24:43.012973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.012876 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" event={"ID":"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0","Type":"ContainerDied","Data":"779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a"} Oct 30 01:24:43.012973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.012937 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs" event={"ID":"593b152c-46a1-4f22-b9b0-bc47a7ad2aa0","Type":"ContainerDied","Data":"2804bcca2548f63b341ea143047263003e7a4a8ba6b789905378ca2da334bcae"} Oct 30 01:24:43.012973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.012962 4905 scope.go:122] "RemoveContainer" containerID="779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a" Oct 30 01:24:43.014687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.014667 4905 generic.go:356] "Generic (PLEG): container finished" podID="b4110fc9-997e-46ed-9968-9b39240e7afb" containerID="a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4" exitCode=0 Oct 30 01:24:43.014876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.014712 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" event={"ID":"b4110fc9-997e-46ed-9968-9b39240e7afb","Type":"ContainerDied","Data":"a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4"} Oct 30 01:24:43.014876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.014746 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" event={"ID":"b4110fc9-997e-46ed-9968-9b39240e7afb","Type":"ContainerDied","Data":"ddd32e6fbab808377cf87450009b617e20b7b1a6c3c5173391899ffdb980fdab"} Oct 30 01:24:43.015010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.014894 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv" Oct 30 01:24:43.027172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.027152 4905 scope.go:122] "RemoveContainer" containerID="779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a" Oct 30 01:24:43.027447 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:43.027423 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a\": container with ID starting with 779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a not found: ID does not exist" containerID="779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a" Oct 30 01:24:43.027491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.027458 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a"} err="failed to get container status \"779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a\": rpc error: code = NotFound desc = could not find container \"779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a\": container with ID starting with 779f4a75dbe4725a0b08e75b35214bf6efda83abd6b52e051b0f4f34f5915f1a not found: ID does not exist" Oct 30 01:24:43.027491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.027481 4905 scope.go:122] "RemoveContainer" containerID="a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4" Oct 30 01:24:43.040043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.040024 4905 scope.go:122] "RemoveContainer" containerID="a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4" Oct 30 01:24:43.040345 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:43.040323 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4\": container with ID starting with a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4 not found: ID does not exist" containerID="a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4" Oct 30 01:24:43.040390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.040351 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4"} err="failed to get container status \"a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4\": rpc error: code = NotFound desc = could not find container \"a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4\": container with ID starting with a72f80485d6077a2e07805cdd566d61fe718dc63e9b451cfe31c09eb1192aeb4 not found: ID does not exist" Oct 30 01:24:43.072563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.072496 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs"] Oct 30 01:24:43.077259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.077233 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cfe8f0-745895f475-bfxxs"] Oct 30 01:24:43.105632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.105574 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv"] Oct 30 01:24:43.109504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.109478 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cfe8f0-7764bbfdf8-5zvnv"] Oct 30 01:24:43.191657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.191626 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" path="/var/lib/kubelet/pods/593b152c-46a1-4f22-b9b0-bc47a7ad2aa0/volumes" Oct 30 01:24:43.192091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:43.192077 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4110fc9-997e-46ed-9968-9b39240e7afb" path="/var/lib/kubelet/pods/b4110fc9-997e-46ed-9968-9b39240e7afb/volumes" Oct 30 01:24:44.831352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:44.831046 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" podUID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:45.785947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:45.785911 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk"] Oct 30 01:24:45.786567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:45.786116 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" containerID="cri-o://ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032" gracePeriod=30 Oct 30 01:24:50.512564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:50.512491 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" podUID="a469becb-c0d0-40cc-b426-3287026d49aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:51.119015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.118983 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:24:51.119407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.119381 4905 generic.go:356] "Generic (PLEG): container finished" podID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerID="ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032" exitCode=0 Oct 30 01:24:51.119492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.119451 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" event={"ID":"90e4567a-0444-4693-a2f0-c832eaf6e6ff","Type":"ContainerDied","Data":"ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032"} Oct 30 01:24:51.119578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.119564 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" event={"ID":"90e4567a-0444-4693-a2f0-c832eaf6e6ff","Type":"ContainerDied","Data":"70437262e6274284276d912ab5474459258a3ba9edeffda97443c89855c2a7cd"} Oct 30 01:24:51.119625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.119598 4905 scope.go:122] "RemoveContainer" containerID="ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032" Oct 30 01:24:51.132827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.132805 4905 scope.go:122] "RemoveContainer" containerID="b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc" Oct 30 01:24:51.145095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.145073 4905 scope.go:122] "RemoveContainer" containerID="ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032" Oct 30 01:24:51.145457 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:51.145434 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032\": container with ID starting with ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032 not found: ID does not exist" containerID="ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032" Oct 30 01:24:51.145568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.145465 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032"} err="failed to get container status \"ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032\": rpc error: code = NotFound desc = could not find container \"ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032\": container with ID starting with ca68b6d6d5634ae4c120507ef1de8030b30c99f5acb56aefcbc93cac20336032 not found: ID does not exist" Oct 30 01:24:51.145568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.145485 4905 scope.go:122] "RemoveContainer" containerID="b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc" Oct 30 01:24:51.145829 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:51.145811 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc\": container with ID starting with b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc not found: ID does not exist" containerID="b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc" Oct 30 01:24:51.145892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.145838 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc"} err="failed to get container status \"b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc\": rpc error: code = NotFound desc = could not find container \"b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc\": container with ID starting with b3586b047d2e9886632d330a267183ceca62346a6318a069c68e247bedf7c8dc not found: ID does not exist" Oct 30 01:24:51.197465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197431 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-envoy\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.197465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197484 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-token\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-token\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.197837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197609 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-kube-api-access-th7s8\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-kube-api-access-th7s8\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.197837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197682 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-certs\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.197837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197714 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-podinfo\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.197837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197740 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-data\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.197837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197793 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-socket\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.198106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.197894 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istiod-ca-cert\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.198106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198059 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-credential-socket\") pod \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\" (UID: \"90e4567a-0444-4693-a2f0-c832eaf6e6ff\") " Oct 30 01:24:51.198793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198442 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-data" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:51.198793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198473 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-socket" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:51.198793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198483 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-certs" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:51.198793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198716 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-credential-socket" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:51.199112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198818 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istiod-ca-cert" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:51.199112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198828 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.199112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198846 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.199112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198856 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.199112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.198864 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.204777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.204750 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-envoy" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:51.205050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.205021 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-token" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:51.206734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.206706 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-podinfo" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:51.207671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.207647 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-kube-api-access-th7s8" pod "90e4567a-0444-4693-a2f0-c832eaf6e6ff" (UID: "90e4567a-0444-4693-a2f0-c832eaf6e6ff"). InnerVolumeSpecName "kube-api-access-th7s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:51.307282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.301090 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g"] Oct 30 01:24:51.307282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.301338 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" containerID="cri-o://00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e" gracePeriod=30 Oct 30 01:24:51.312393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.310922 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9"] Oct 30 01:24:51.340587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.336352 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.340587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.336380 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.340587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.336413 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7s8\" (UniqueName: \"kubernetes.io/projected/90e4567a-0444-4693-a2f0-c832eaf6e6ff-kube-api-access-th7s8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.340587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.336426 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.340587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.336439 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e4567a-0444-4693-a2f0-c832eaf6e6ff-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.539676 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:51.539629 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" podUID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:52.127716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:52.127679 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk" Oct 30 01:24:52.184292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:52.184261 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk"] Oct 30 01:24:52.197731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:52.197701 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-2g9nk"] Oct 30 01:24:52.858767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:52.858723 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" podUID="4931e93a-0f20-47c1-9f73-ffe763e183b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:53.218338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.216973 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" path="/var/lib/kubelet/pods/90e4567a-0444-4693-a2f0-c832eaf6e6ff/volumes" Oct 30 01:24:53.680842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.680810 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554"] Oct 30 01:24:53.681531 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681507 4905 cpu_manager.go:414] "containerMap: removing container" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" Oct 30 01:24:53.681582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681534 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" Oct 30 01:24:53.681582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681545 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" Oct 30 01:24:53.681582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681549 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" Oct 30 01:24:53.681582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681564 4905 cpu_manager.go:414] "containerMap: removing container" podUID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" containerName="payload-pre-processing" Oct 30 01:24:53.681582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681569 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" containerName="payload-pre-processing" Oct 30 01:24:53.681582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681583 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681588 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681597 4905 cpu_manager.go:414] "containerMap: removing container" podUID="407de83a-8478-47d5-8fce-4ea9509263bb" containerName="payload-pre-processing" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681602 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407de83a-8478-47d5-8fce-4ea9509263bb" containerName="payload-pre-processing" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681610 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b4110fc9-997e-46ed-9968-9b39240e7afb" containerName="payload-processing" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681614 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4110fc9-997e-46ed-9968-9b39240e7afb" containerName="payload-processing" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681619 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681623 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681634 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681637 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681645 4905 cpu_manager.go:414] "containerMap: removing container" podUID="dc1d798f-f790-41d3-b9f0-773cf307b368" containerName="maas-api" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681649 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1d798f-f790-41d3-b9f0-773cf307b368" containerName="maas-api" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681656 4905 cpu_manager.go:414] "containerMap: removing container" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681661 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681668 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681673 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681679 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" containerName="payload-pre-processing" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681684 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" containerName="payload-pre-processing" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681695 4905 cpu_manager.go:414] "containerMap: removing container" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681700 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681708 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681712 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681723 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681727 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681737 4905 cpu_manager.go:414] "containerMap: removing container" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681741 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681756 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681760 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681767 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" Oct 30 01:24:53.681779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681772 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681864 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="593b152c-46a1-4f22-b9b0-bc47a7ad2aa0" containerName="payload-pre-processing" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681874 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681880 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="407de83a-8478-47d5-8fce-4ea9509263bb" containerName="payload-pre-processing" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681887 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4110fc9-997e-46ed-9968-9b39240e7afb" containerName="payload-processing" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681893 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681899 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681904 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1d798f-f790-41d3-b9f0-773cf307b368" containerName="maas-api" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681910 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681917 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681922 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3399f18d-1f37-4f43-8b91-0bfff4b22e1d" containerName="payload-pre-processing" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681928 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e4567a-0444-4693-a2f0-c832eaf6e6ff" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681935 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681941 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681948 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44d7288-6855-4519-be9e-ab84d52bbb57" containerName="istio-proxy" Oct 30 01:24:53.682756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.681955 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ccc4f5-f71a-4ca8-9bb2-eb34a4eb2290" containerName="istio-proxy" Oct 30 01:24:53.687442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.687421 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:53.704974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.704950 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554"] Oct 30 01:24:53.714266 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.714243 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-11e7a9\"" Oct 30 01:24:53.714414 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.714375 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-11e7a9-dockercfg-d6z7m\"" Oct 30 01:24:53.800500 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.800462 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:53.800700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.800674 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkjg\" (UniqueName: \"kubernetes.io/projected/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-kube-api-access-fnkjg\") pod \"payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:53.831867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.831829 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6"] Oct 30 01:24:53.832662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.832639 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" Oct 30 01:24:53.832662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.832661 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" Oct 30 01:24:53.832850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.832813 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf339a-6d1f-460e-8b17-57b3494d6707" containerName="istio-proxy" Oct 30 01:24:53.832850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.832831 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9370e3cd-a5c6-4b32-833b-68195297a32f" containerName="istio-proxy" Oct 30 01:24:53.832961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.832858 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cde45c-ce1c-4b18-81c7-a0edb5afd13a" containerName="istio-proxy" Oct 30 01:24:53.832961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.832870 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6733ee-c5e2-4d99-8c1b-4d2a7077b76a" containerName="istio-proxy" Oct 30 01:24:53.837022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.836612 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:53.856258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.856231 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6"] Oct 30 01:24:53.902168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.902133 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkjg\" (UniqueName: \"kubernetes.io/projected/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-kube-api-access-fnkjg\") pod \"payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:53.902679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.902309 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:53.903211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.903183 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:53.922919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:53.922892 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkjg\" (UniqueName: \"kubernetes.io/projected/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-kube-api-access-fnkjg\") pod \"payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:54.002492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.002442 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:54.003193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.003167 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdp9x\" (UniqueName: \"kubernetes.io/projected/e3b9fab1-468c-451d-b380-4870b39a6bad-kube-api-access-pdp9x\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:54.003440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.003420 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b9fab1-468c-451d-b380-4870b39a6bad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:54.104416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.104380 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b9fab1-468c-451d-b380-4870b39a6bad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:54.104622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.104501 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdp9x\" (UniqueName: \"kubernetes.io/projected/e3b9fab1-468c-451d-b380-4870b39a6bad-kube-api-access-pdp9x\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:54.105226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.105201 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b9fab1-468c-451d-b380-4870b39a6bad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:54.140578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.125111 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdp9x\" (UniqueName: \"kubernetes.io/projected/e3b9fab1-468c-451d-b380-4870b39a6bad-kube-api-access-pdp9x\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:54.180626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.165413 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:54.187020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.186984 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd"] Oct 30 01:24:54.187397 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.187366 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" podUID="6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" containerName="maas-api" containerID="cri-o://4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4" gracePeriod=30 Oct 30 01:24:54.320646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.320613 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554"] Oct 30 01:24:54.321032 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:24:54.321013 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc48de16_a096_4ab3_aa7b_d15520e4ed6b.slice/crio-6540fe1126795eefab3646d41a5ddcbb7fb59e3ceb8c45d270c21882195daf48 WatchSource:0}: Error finding container 6540fe1126795eefab3646d41a5ddcbb7fb59e3ceb8c45d270c21882195daf48: Status 404 returned error can't find the container with id 6540fe1126795eefab3646d41a5ddcbb7fb59e3ceb8c45d270c21882195daf48 Oct 30 01:24:54.638858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.638805 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6"] Oct 30 01:24:54.704924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.696349 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97"] Oct 30 01:24:54.704924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.696750 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" podUID="ede6ca67-abbc-4bda-8b14-2f3ac1eac173" containerName="maas-api" containerID="cri-o://a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f" gracePeriod=30 Oct 30 01:24:54.787640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.786802 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" podUID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:54.821555 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:54.821429 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede6ca67_abbc_4bda_8b14_2f3ac1eac173.slice/crio-conmon-a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede6ca67_abbc_4bda_8b14_2f3ac1eac173.slice/crio-a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:24:54.844271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.843628 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:24:54.924935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.923091 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-metrics-certs\") pod \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " Oct 30 01:24:54.924935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.923228 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-maas-api-tls\") pod \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " Oct 30 01:24:54.924935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.923278 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-kube-api-access-z5hbm\" (UniqueName: \"kubernetes.io/projected/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-kube-api-access-z5hbm\") pod \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " Oct 30 01:24:54.924935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.923338 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-openshift-service-ca\") pod \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\" (UID: \"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea\") " Oct 30 01:24:54.925838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.925752 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-openshift-service-ca" pod "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" (UID: "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:54.928298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.928209 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-maas-api-tls" pod "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" (UID: "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:54.928298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.928263 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-metrics-certs" pod "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" (UID: "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:54.930207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:54.930172 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-kube-api-access-z5hbm" pod "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" (UID: "6a2b4abf-aafe-4c3f-a52d-d05862efb7ea"). InnerVolumeSpecName "kube-api-access-z5hbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:55.027770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.027725 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.027770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.027761 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.028011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.027776 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5hbm\" (UniqueName: \"kubernetes.io/projected/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-kube-api-access-z5hbm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.028011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.027790 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.146065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.145740 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:24:55.233074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.230391 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-kube-api-access-fw2cx\" (UniqueName: \"kubernetes.io/projected/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-kube-api-access-fw2cx\") pod \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " Oct 30 01:24:55.233074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.230461 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-metrics-certs\") pod \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " Oct 30 01:24:55.233074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.230731 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-openshift-service-ca\") pod \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " Oct 30 01:24:55.233074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.230797 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-maas-api-tls\") pod \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\" (UID: \"ede6ca67-abbc-4bda-8b14-2f3ac1eac173\") " Oct 30 01:24:55.233074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.232697 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-openshift-service-ca" pod "ede6ca67-abbc-4bda-8b14-2f3ac1eac173" (UID: "ede6ca67-abbc-4bda-8b14-2f3ac1eac173"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:55.359736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.335559 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.359736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.346115 4905 generic.go:356] "Generic (PLEG): container finished" podID="ede6ca67-abbc-4bda-8b14-2f3ac1eac173" containerID="a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f" exitCode=0 Oct 30 01:24:55.359736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.346323 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" Oct 30 01:24:55.360436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.360408 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-kube-api-access-fw2cx" pod "ede6ca67-abbc-4bda-8b14-2f3ac1eac173" (UID: "ede6ca67-abbc-4bda-8b14-2f3ac1eac173"). InnerVolumeSpecName "kube-api-access-fw2cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:55.360711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.360693 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-metrics-certs" pod "ede6ca67-abbc-4bda-8b14-2f3ac1eac173" (UID: "ede6ca67-abbc-4bda-8b14-2f3ac1eac173"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:55.380619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.378415 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" event={"ID":"ede6ca67-abbc-4bda-8b14-2f3ac1eac173","Type":"ContainerDied","Data":"a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f"} Oct 30 01:24:55.380619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.378476 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97" event={"ID":"ede6ca67-abbc-4bda-8b14-2f3ac1eac173","Type":"ContainerDied","Data":"3b124f2ffab6abc2f81e5644f6ed97d846af46d28ecff2fbb32e00e53aa364b3"} Oct 30 01:24:55.380619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.378507 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:24:55.380619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.378553 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" event={"ID":"dc48de16-a096-4ab3-aa7b-d15520e4ed6b","Type":"ContainerStarted","Data":"7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8"} Oct 30 01:24:55.380619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.378575 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" event={"ID":"dc48de16-a096-4ab3-aa7b-d15520e4ed6b","Type":"ContainerStarted","Data":"6540fe1126795eefab3646d41a5ddcbb7fb59e3ceb8c45d270c21882195daf48"} Oct 30 01:24:55.385780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.381978 4905 scope.go:122] "RemoveContainer" containerID="a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f" Oct 30 01:24:55.387118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.387092 4905 generic.go:356] "Generic (PLEG): container finished" podID="6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" containerID="4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4" exitCode=0 Oct 30 01:24:55.387276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.387258 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" event={"ID":"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea","Type":"ContainerDied","Data":"4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4"} Oct 30 01:24:55.387343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.387300 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" event={"ID":"6a2b4abf-aafe-4c3f-a52d-d05862efb7ea","Type":"ContainerDied","Data":"509ca50f1423bf4162403f6d22eb929e4921520d04441bc7e457b29a8b2b5d3e"} Oct 30 01:24:55.387395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.387350 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd" Oct 30 01:24:55.392585 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.391917 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" event={"ID":"e3b9fab1-468c-451d-b380-4870b39a6bad","Type":"ContainerStarted","Data":"d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf"} Oct 30 01:24:55.392585 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.391962 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" event={"ID":"e3b9fab1-468c-451d-b380-4870b39a6bad","Type":"ContainerStarted","Data":"108cb377fd60a9d7463e1a403febfeb52698e63019822a7d65d2ba5ae9f024c5"} Oct 30 01:24:55.392935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.392837 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:24:55.395861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.395835 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-maas-api-tls" pod "ede6ca67-abbc-4bda-8b14-2f3ac1eac173" (UID: "ede6ca67-abbc-4bda-8b14-2f3ac1eac173"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:55.404061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.404040 4905 scope.go:122] "RemoveContainer" containerID="a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f" Oct 30 01:24:55.404362 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:55.404343 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f\": container with ID starting with a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f not found: ID does not exist" containerID="a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f" Oct 30 01:24:55.404428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.404368 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f"} err="failed to get container status \"a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f\": rpc error: code = NotFound desc = could not find container \"a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f\": container with ID starting with a893e3cabb9ee4dfdf6fe2516842c5a0034f6ef232235fabc497c81ba010a84f not found: ID does not exist" Oct 30 01:24:55.404428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.404383 4905 scope.go:122] "RemoveContainer" containerID="4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4" Oct 30 01:24:55.420158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.420132 4905 scope.go:122] "RemoveContainer" containerID="4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4" Oct 30 01:24:55.420676 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:55.420644 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4\": container with ID starting with 4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4 not found: ID does not exist" containerID="4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4" Oct 30 01:24:55.420787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.420684 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4"} err="failed to get container status \"4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4\": rpc error: code = NotFound desc = could not find container \"4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4\": container with ID starting with 4c8c3bb0020cb6e15d35f985d1573fefb3652775afcf10760e9b2f30ccd50be4 not found: ID does not exist" Oct 30 01:24:55.421152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.421108 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" podStartSLOduration=2.421096668 podStartE2EDuration="2.421096668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:55.418797301 +0000 UTC m=+2780.361970496" watchObservedRunningTime="2026-10-30 01:24:55.421096668 +0000 UTC m=+2780.364269860" Oct 30 01:24:55.437310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.437277 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.438750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.438734 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2cx\" (UniqueName: \"kubernetes.io/projected/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-kube-api-access-fw2cx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.438903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.438890 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede6ca67-abbc-4bda-8b14-2f3ac1eac173-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:55.478103 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.478078 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd"] Oct 30 01:24:55.481629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.481607 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d0ae0d-867549d69b-8g6rd"] Oct 30 01:24:55.517865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.517822 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" podStartSLOduration=2.517808435 podStartE2EDuration="2.517808435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:55.517026076 +0000 UTC m=+2780.460199270" watchObservedRunningTime="2026-10-30 01:24:55.517808435 +0000 UTC m=+2780.460981628" Oct 30 01:24:55.743599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.743564 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97"] Oct 30 01:24:55.748699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:55.748677 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bdf6f3-7db8b56f4d-pvx97"] Oct 30 01:24:56.647955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.647927 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:24:56.657647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.657621 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-kube-api-access-klc6n\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-kube-api-access-klc6n\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.658398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.658346 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-socket\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.658754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.658717 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17922593-835e-4bd3-9e76-7510d501815f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17922593-835e-4bd3-9e76-7510d501815f-istiod-ca-cert\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.658991 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.658496 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-socket" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:56.659130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.659113 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-data\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.673057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.659128 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17922593-835e-4bd3-9e76-7510d501815f-istiod-ca-cert" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:56.673057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.659377 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-data" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:56.673057 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.661841 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-kube-api-access-klc6n" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "kube-api-access-klc6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:56.673362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.673344 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17922593-835e-4bd3-9e76-7510d501815f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17922593-835e-4bd3-9e76-7510d501815f-istio-podinfo\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.673548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.673505 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-credential-socket\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.673643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.673565 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-certs\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.673696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.673680 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-envoy\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.674197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.674172 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-certs" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:56.674569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.674544 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-credential-socket" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:56.674739 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.674722 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-istio-token\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-istio-token\") pod \"17922593-835e-4bd3-9e76-7510d501815f\" (UID: \"17922593-835e-4bd3-9e76-7510d501815f\") " Oct 30 01:24:56.675556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.675480 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klc6n\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-kube-api-access-klc6n\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.675690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.675676 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.675779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.675769 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17922593-835e-4bd3-9e76-7510d501815f-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.675869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.675859 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.675951 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.675941 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.676031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.676019 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.676805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.676784 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17922593-835e-4bd3-9e76-7510d501815f-istio-podinfo" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:56.676942 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.676919 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-envoy" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:56.678704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.678682 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-istio-token" pod "17922593-835e-4bd3-9e76-7510d501815f" (UID: "17922593-835e-4bd3-9e76-7510d501815f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:56.791551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.791484 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17922593-835e-4bd3-9e76-7510d501815f-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.791706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.791577 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17922593-835e-4bd3-9e76-7510d501815f-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:56.791706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:56.791595 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17922593-835e-4bd3-9e76-7510d501815f-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:24:57.225850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.225777 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" path="/var/lib/kubelet/pods/6a2b4abf-aafe-4c3f-a52d-d05862efb7ea/volumes" Oct 30 01:24:57.226588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.226553 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede6ca67-abbc-4bda-8b14-2f3ac1eac173" path="/var/lib/kubelet/pods/ede6ca67-abbc-4bda-8b14-2f3ac1eac173/volumes" Oct 30 01:24:57.416693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.416660 4905 generic.go:356] "Generic (PLEG): container finished" podID="17922593-835e-4bd3-9e76-7510d501815f" containerID="00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e" exitCode=0 Oct 30 01:24:57.416913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.416864 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" event={"ID":"17922593-835e-4bd3-9e76-7510d501815f","Type":"ContainerDied","Data":"00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e"} Oct 30 01:24:57.417049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.416914 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" event={"ID":"17922593-835e-4bd3-9e76-7510d501815f","Type":"ContainerDied","Data":"a8f7657138d549ab042de8e4dea3005376538df3259a1b46fdb430c23c3388e3"} Oct 30 01:24:57.417049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.416938 4905 scope.go:122] "RemoveContainer" containerID="00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e" Oct 30 01:24:57.417049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.416994 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g" Oct 30 01:24:57.449077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.448716 4905 scope.go:122] "RemoveContainer" containerID="d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c" Oct 30 01:24:57.465622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.465013 4905 scope.go:122] "RemoveContainer" containerID="00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e" Oct 30 01:24:57.466393 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:57.466363 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e\": container with ID starting with 00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e not found: ID does not exist" containerID="00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e" Oct 30 01:24:57.466492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.466405 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e"} err="failed to get container status \"00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e\": rpc error: code = NotFound desc = could not find container \"00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e\": container with ID starting with 00fe45fca06e907fa4b8a1d9a0a1c3b554e36127f7db89eb1145b84b25b2cf1e not found: ID does not exist" Oct 30 01:24:57.466492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.466428 4905 scope.go:122] "RemoveContainer" containerID="d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c" Oct 30 01:24:57.466840 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:24:57.466815 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c\": container with ID starting with d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c not found: ID does not exist" containerID="d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c" Oct 30 01:24:57.466902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.466852 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c"} err="failed to get container status \"d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c\": rpc error: code = NotFound desc = could not find container \"d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c\": container with ID starting with d10feb8a7d176408146e1ea2f808a53f16fb5880364575c4deb7e6f9ae2f947c not found: ID does not exist" Oct 30 01:24:57.491205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.491175 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g"] Oct 30 01:24:57.504950 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:57.504927 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-tbd4g"] Oct 30 01:24:59.193413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:24:59.193380 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17922593-835e-4bd3-9e76-7510d501815f" path="/var/lib/kubelet/pods/17922593-835e-4bd3-9e76-7510d501815f/volumes" Oct 30 01:25:00.183911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.158823 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:25:00.260625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.258383 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:25:00.268990 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.268959 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd181bda-aa31-4f73-9a62-c3485fa03d1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd181bda-aa31-4f73-9a62-c3485fa03d1f-plugins-config-volume\") pod \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " Oct 30 01:25:00.269903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.269863 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd181bda-aa31-4f73-9a62-c3485fa03d1f-kube-api-access-vb6w6\" (UniqueName: \"kubernetes.io/projected/fd181bda-aa31-4f73-9a62-c3485fa03d1f-kube-api-access-vb6w6\") pod \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\" (UID: \"fd181bda-aa31-4f73-9a62-c3485fa03d1f\") " Oct 30 01:25:00.270379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.269545 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd181bda-aa31-4f73-9a62-c3485fa03d1f-plugins-config-volume" pod "fd181bda-aa31-4f73-9a62-c3485fa03d1f" (UID: "fd181bda-aa31-4f73-9a62-c3485fa03d1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:00.279589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.277145 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd181bda-aa31-4f73-9a62-c3485fa03d1f-kube-api-access-vb6w6" pod "fd181bda-aa31-4f73-9a62-c3485fa03d1f" (UID: "fd181bda-aa31-4f73-9a62-c3485fa03d1f"). InnerVolumeSpecName "kube-api-access-vb6w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:00.371879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.371849 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a469becb-c0d0-40cc-b426-3287026d49aa-kube-api-access-dlsjp\" (UniqueName: \"kubernetes.io/projected/a469becb-c0d0-40cc-b426-3287026d49aa-kube-api-access-dlsjp\") pod \"a469becb-c0d0-40cc-b426-3287026d49aa\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " Oct 30 01:25:00.372032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.371933 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a469becb-c0d0-40cc-b426-3287026d49aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a469becb-c0d0-40cc-b426-3287026d49aa-plugins-config-volume\") pod \"a469becb-c0d0-40cc-b426-3287026d49aa\" (UID: \"a469becb-c0d0-40cc-b426-3287026d49aa\") " Oct 30 01:25:00.372317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.372295 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a469becb-c0d0-40cc-b426-3287026d49aa-plugins-config-volume" pod "a469becb-c0d0-40cc-b426-3287026d49aa" (UID: "a469becb-c0d0-40cc-b426-3287026d49aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:00.372575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.372554 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a469becb-c0d0-40cc-b426-3287026d49aa-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:00.372575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.372573 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd181bda-aa31-4f73-9a62-c3485fa03d1f-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:00.372679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.372585 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6w6\" (UniqueName: \"kubernetes.io/projected/fd181bda-aa31-4f73-9a62-c3485fa03d1f-kube-api-access-vb6w6\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:00.374207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.374180 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a469becb-c0d0-40cc-b426-3287026d49aa-kube-api-access-dlsjp" pod "a469becb-c0d0-40cc-b426-3287026d49aa" (UID: "a469becb-c0d0-40cc-b426-3287026d49aa"). InnerVolumeSpecName "kube-api-access-dlsjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:00.481551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.480191 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" Oct 30 01:25:00.481551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.480279 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" event={"ID":"fd181bda-aa31-4f73-9a62-c3485fa03d1f","Type":"ContainerDied","Data":"15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce"} Oct 30 01:25:00.481551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.480369 4905 scope.go:122] "RemoveContainer" containerID="15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce" Oct 30 01:25:00.481551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.479940 4905 generic.go:356] "Generic (PLEG): container finished" podID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" containerID="15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce" exitCode=0 Oct 30 01:25:00.481551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.481389 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f" event={"ID":"fd181bda-aa31-4f73-9a62-c3485fa03d1f","Type":"ContainerDied","Data":"c8f7a1baa99fcb0cfe8c1d57ceedd2e662eb92ca71f5459fb5cf86d976b1efd7"} Oct 30 01:25:00.502544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.500104 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlsjp\" (UniqueName: \"kubernetes.io/projected/a469becb-c0d0-40cc-b426-3287026d49aa-kube-api-access-dlsjp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:00.544577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.536420 4905 generic.go:356] "Generic (PLEG): container finished" podID="a469becb-c0d0-40cc-b426-3287026d49aa" containerID="bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a" exitCode=0 Oct 30 01:25:00.544577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.536586 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" Oct 30 01:25:00.544577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.536609 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" event={"ID":"a469becb-c0d0-40cc-b426-3287026d49aa","Type":"ContainerDied","Data":"bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a"} Oct 30 01:25:00.544577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.537431 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq" event={"ID":"a469becb-c0d0-40cc-b426-3287026d49aa","Type":"ContainerDied","Data":"fa7c032cebba632e3bb029ac8d6763b5d55419f772d061fc3b283cdd2900ed37"} Oct 30 01:25:00.563733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.562729 4905 scope.go:122] "RemoveContainer" containerID="15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce" Oct 30 01:25:00.565638 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:00.565169 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce\": container with ID starting with 15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce not found: ID does not exist" containerID="15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce" Oct 30 01:25:00.565638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.565209 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce"} err="failed to get container status \"15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce\": rpc error: code = NotFound desc = could not find container \"15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce\": container with ID starting with 15f971de86184a0939a9c3db86253e0ff733280ce0be0cd2ed3032f017bb0fce not found: ID does not exist" Oct 30 01:25:00.565638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.565237 4905 scope.go:122] "RemoveContainer" containerID="bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a" Oct 30 01:25:00.582055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.582030 4905 scope.go:122] "RemoveContainer" containerID="bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a" Oct 30 01:25:00.582910 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:00.582873 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a\": container with ID starting with bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a not found: ID does not exist" containerID="bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a" Oct 30 01:25:00.583029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.582921 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a"} err="failed to get container status \"bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a\": rpc error: code = NotFound desc = could not find container \"bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a\": container with ID starting with bb8fc882196667c1bd27b1a3ea6be33a56dd9686dfb62444fd7dfed4e57a2f0a not found: ID does not exist" Oct 30 01:25:00.616307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.616276 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f"] Oct 30 01:25:00.620204 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.620115 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bdf6f3-65bd7fb9bb-9xh5f"] Oct 30 01:25:00.656500 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.656470 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq"] Oct 30 01:25:00.659933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:00.659907 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bdf6f3-68f89d7779-sgktq"] Oct 30 01:25:01.194946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:01.194911 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a469becb-c0d0-40cc-b426-3287026d49aa" path="/var/lib/kubelet/pods/a469becb-c0d0-40cc-b426-3287026d49aa/volumes" Oct 30 01:25:01.195466 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:01.195444 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" path="/var/lib/kubelet/pods/fd181bda-aa31-4f73-9a62-c3485fa03d1f/volumes" Oct 30 01:25:02.369628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.367846 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:25:02.468201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.468151 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-kube-api-access-6c9hp\" (UniqueName: \"kubernetes.io/projected/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-kube-api-access-6c9hp\") pod \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " Oct 30 01:25:02.468410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.468270 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-plugins-config-volume\") pod \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\" (UID: \"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e\") " Oct 30 01:25:02.469330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.469271 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-plugins-config-volume" pod "7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" (UID: "7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:02.472428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.472391 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-kube-api-access-6c9hp" pod "7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" (UID: "7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e"). InnerVolumeSpecName "kube-api-access-6c9hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:02.512591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.505689 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4"] Oct 30 01:25:02.512591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.505965 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" containerID="cri-o://2c87d657eeb30879083c1231814cb6a9ff9952a47827af8fdffa072bc747d921" gracePeriod=30 Oct 30 01:25:02.571577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.571470 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9hp\" (UniqueName: \"kubernetes.io/projected/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-kube-api-access-6c9hp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:02.571577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.571511 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:02.590284 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.590254 4905 generic.go:356] "Generic (PLEG): container finished" podID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" containerID="49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4" exitCode=0 Oct 30 01:25:02.590460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.590323 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" Oct 30 01:25:02.590460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.590357 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" event={"ID":"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e","Type":"ContainerDied","Data":"49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4"} Oct 30 01:25:02.590460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.590416 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67" event={"ID":"7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e","Type":"ContainerDied","Data":"f8cb959539addb1ead4650df739a0bbc182763d7de40224a4f7a07bf1b82c8d8"} Oct 30 01:25:02.590460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.590440 4905 scope.go:122] "RemoveContainer" containerID="49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4" Oct 30 01:25:02.601703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.601635 4905 scope.go:122] "RemoveContainer" containerID="49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4" Oct 30 01:25:02.601961 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:02.601936 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4\": container with ID starting with 49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4 not found: ID does not exist" containerID="49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4" Oct 30 01:25:02.602067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.601972 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4"} err="failed to get container status \"49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4\": rpc error: code = NotFound desc = could not find container \"49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4\": container with ID starting with 49cedfe467a4778b897d39f20b81614284ea2bbf8b5d707b9594f4840434b1c4 not found: ID does not exist" Oct 30 01:25:02.749602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.749562 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67"] Oct 30 01:25:02.749844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.749833 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b973b-5db798b9bc-smv67"] Oct 30 01:25:02.878064 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:02.877975 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" podUID="4931e93a-0f20-47c1-9f73-ffe763e183b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:03.266567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.258502 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" path="/var/lib/kubelet/pods/7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e/volumes" Oct 30 01:25:03.272110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.272057 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n"] Oct 30 01:25:03.272615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.272570 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" containerID="cri-o://6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343" gracePeriod=30 Oct 30 01:25:03.522542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.521791 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:25:03.600540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.599217 4905 generic.go:356] "Generic (PLEG): container finished" podID="4931e93a-0f20-47c1-9f73-ffe763e183b1" containerID="e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458" exitCode=0 Oct 30 01:25:03.600540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.599433 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" event={"ID":"4931e93a-0f20-47c1-9f73-ffe763e183b1","Type":"ContainerDied","Data":"e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458"} Oct 30 01:25:03.600540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.599478 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" event={"ID":"4931e93a-0f20-47c1-9f73-ffe763e183b1","Type":"ContainerDied","Data":"500d157b0792090b35c8125a4288a88dfd717cd50cfa70249165106432d071c6"} Oct 30 01:25:03.600540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.599499 4905 scope.go:122] "RemoveContainer" containerID="e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458" Oct 30 01:25:03.600540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.599620 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk" Oct 30 01:25:03.612047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.611995 4905 scope.go:122] "RemoveContainer" containerID="e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458" Oct 30 01:25:03.612269 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:03.612250 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458\": container with ID starting with e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458 not found: ID does not exist" containerID="e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458" Oct 30 01:25:03.612361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.612283 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458"} err="failed to get container status \"e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458\": rpc error: code = NotFound desc = could not find container \"e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458\": container with ID starting with e2e72d3da21916bdbd5d6dc9414aad203df2724256dde8d8631eda259d790458 not found: ID does not exist" Oct 30 01:25:03.619158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.619137 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4931e93a-0f20-47c1-9f73-ffe763e183b1-kube-api-access-nknpz\" (UniqueName: \"kubernetes.io/projected/4931e93a-0f20-47c1-9f73-ffe763e183b1-kube-api-access-nknpz\") pod \"4931e93a-0f20-47c1-9f73-ffe763e183b1\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " Oct 30 01:25:03.619472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.619447 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4931e93a-0f20-47c1-9f73-ffe763e183b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4931e93a-0f20-47c1-9f73-ffe763e183b1-plugins-config-volume\") pod \"4931e93a-0f20-47c1-9f73-ffe763e183b1\" (UID: \"4931e93a-0f20-47c1-9f73-ffe763e183b1\") " Oct 30 01:25:03.619918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.619884 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4931e93a-0f20-47c1-9f73-ffe763e183b1-plugins-config-volume" pod "4931e93a-0f20-47c1-9f73-ffe763e183b1" (UID: "4931e93a-0f20-47c1-9f73-ffe763e183b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:03.620206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.620189 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4931e93a-0f20-47c1-9f73-ffe763e183b1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.621821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.621801 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4931e93a-0f20-47c1-9f73-ffe763e183b1-kube-api-access-nknpz" pod "4931e93a-0f20-47c1-9f73-ffe763e183b1" (UID: "4931e93a-0f20-47c1-9f73-ffe763e183b1"). InnerVolumeSpecName "kube-api-access-nknpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:03.721726 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.721650 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nknpz\" (UniqueName: \"kubernetes.io/projected/4931e93a-0f20-47c1-9f73-ffe763e183b1-kube-api-access-nknpz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.985615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.985512 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk"] Oct 30 01:25:03.998563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:03.998530 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d0ae0d-7c89c4947-822wk"] Oct 30 01:25:04.499442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:04.499402 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d"] Oct 30 01:25:04.545213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:04.545172 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p"] Oct 30 01:25:05.317670 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:05.317600 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4931e93a-0f20-47c1-9f73-ffe763e183b1" path="/var/lib/kubelet/pods/4931e93a-0f20-47c1-9f73-ffe763e183b1/volumes" Oct 30 01:25:06.382059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.382017 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" containerID="cri-o://c432493aa78825acdbedf15ed643774c4f03fbe8a0bd6f82d92b99e1cd2db135" gracePeriod=45 Oct 30 01:25:06.404107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.404088 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:25:06.404426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.404250 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:25:06.634187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.633879 4905 generic.go:356] "Generic (PLEG): container finished" podID="5f264934-e2ee-435e-898f-921f20b00f16" containerID="c432493aa78825acdbedf15ed643774c4f03fbe8a0bd6f82d92b99e1cd2db135" exitCode=0 Oct 30 01:25:06.634187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.634075 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" event={"ID":"5f264934-e2ee-435e-898f-921f20b00f16","Type":"ContainerDied","Data":"c432493aa78825acdbedf15ed643774c4f03fbe8a0bd6f82d92b99e1cd2db135"} Oct 30 01:25:06.739217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.739180 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:25:06.817511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.817472 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-tmp-dir\") pod \"5f264934-e2ee-435e-898f-921f20b00f16\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " Oct 30 01:25:06.817806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.817659 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f264934-e2ee-435e-898f-921f20b00f16-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f264934-e2ee-435e-898f-921f20b00f16-tls-certs\") pod \"5f264934-e2ee-435e-898f-921f20b00f16\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " Oct 30 01:25:06.817806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.817708 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-model-cache\") pod \"5f264934-e2ee-435e-898f-921f20b00f16\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " Oct 30 01:25:06.817925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.817817 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f264934-e2ee-435e-898f-921f20b00f16-kube-api-access-57x6r\" (UniqueName: \"kubernetes.io/projected/5f264934-e2ee-435e-898f-921f20b00f16-kube-api-access-57x6r\") pod \"5f264934-e2ee-435e-898f-921f20b00f16\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " Oct 30 01:25:06.817925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.817860 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-home\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-home\") pod \"5f264934-e2ee-435e-898f-921f20b00f16\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " Oct 30 01:25:06.818041 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.818008 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-dshm\") pod \"5f264934-e2ee-435e-898f-921f20b00f16\" (UID: \"5f264934-e2ee-435e-898f-921f20b00f16\") " Oct 30 01:25:06.819313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.819270 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-home" pod "5f264934-e2ee-435e-898f-921f20b00f16" (UID: "5f264934-e2ee-435e-898f-921f20b00f16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:06.819702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.819681 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-tmp-dir" pod "5f264934-e2ee-435e-898f-921f20b00f16" (UID: "5f264934-e2ee-435e-898f-921f20b00f16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:06.819936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.819889 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-model-cache" pod "5f264934-e2ee-435e-898f-921f20b00f16" (UID: "5f264934-e2ee-435e-898f-921f20b00f16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:06.822984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.822953 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-dshm" pod "5f264934-e2ee-435e-898f-921f20b00f16" (UID: "5f264934-e2ee-435e-898f-921f20b00f16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:06.826567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.825564 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f264934-e2ee-435e-898f-921f20b00f16-kube-api-access-57x6r" pod "5f264934-e2ee-435e-898f-921f20b00f16" (UID: "5f264934-e2ee-435e-898f-921f20b00f16"). InnerVolumeSpecName "kube-api-access-57x6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:06.826567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.825868 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f264934-e2ee-435e-898f-921f20b00f16-tls-certs" pod "5f264934-e2ee-435e-898f-921f20b00f16" (UID: "5f264934-e2ee-435e-898f-921f20b00f16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:06.919918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.919845 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:06.919918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.919870 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:06.919918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.919880 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f264934-e2ee-435e-898f-921f20b00f16-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:06.919918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.919888 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:06.919918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.919896 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57x6r\" (UniqueName: \"kubernetes.io/projected/5f264934-e2ee-435e-898f-921f20b00f16-kube-api-access-57x6r\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:06.919918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:06.919904 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f264934-e2ee-435e-898f-921f20b00f16-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:07.645669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.645628 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" event={"ID":"5f264934-e2ee-435e-898f-921f20b00f16","Type":"ContainerDied","Data":"81be80d14b9f0f765045f1a69711df07a3ff1b5ca4484238a92b85154578471a"} Oct 30 01:25:07.646155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.645691 4905 scope.go:122] "RemoveContainer" containerID="c432493aa78825acdbedf15ed643774c4f03fbe8a0bd6f82d92b99e1cd2db135" Oct 30 01:25:07.646155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.645827 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" Oct 30 01:25:07.650006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.649973 4905 generic.go:356] "Generic (PLEG): container finished" podID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerID="2c87d657eeb30879083c1231814cb6a9ff9952a47827af8fdffa072bc747d921" exitCode=0 Oct 30 01:25:07.650142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.650049 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" event={"ID":"70491b5d-60ce-4630-997e-9f16b05bc8b4","Type":"ContainerDied","Data":"2c87d657eeb30879083c1231814cb6a9ff9952a47827af8fdffa072bc747d921"} Oct 30 01:25:07.659130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.659093 4905 scope.go:122] "RemoveContainer" containerID="f474304491c598d8f17f6176621622605723fd495ae5aceb845b69ea8be1f0f1" Oct 30 01:25:07.706096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.706067 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9"] Oct 30 01:25:07.706271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.706156 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:25:07.718793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.718764 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/premium-w3-298ddc-kserve-86b6b4d799-25ff9"] Oct 30 01:25:07.862546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.862283 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:25:07.938059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938018 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-envoy\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938103 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-kube-api-access-nqstz\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-kube-api-access-nqstz\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938162 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-data\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938215 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-podinfo\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938295 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-credential-socket\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938378 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-certs\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938418 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-socket\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938536 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-token\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-token\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.938684 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.938576 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70491b5d-60ce-4630-997e-9f16b05bc8b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70491b5d-60ce-4630-997e-9f16b05bc8b4-istiod-ca-cert\") pod \"70491b5d-60ce-4630-997e-9f16b05bc8b4\" (UID: \"70491b5d-60ce-4630-997e-9f16b05bc8b4\") " Oct 30 01:25:07.939507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.939481 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70491b5d-60ce-4630-997e-9f16b05bc8b4-istiod-ca-cert" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:07.940075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.940048 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-credential-socket" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:07.940248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.940124 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-socket" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:07.940332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.940282 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-certs" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:07.942062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.941002 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-data" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:07.944181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.944141 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-podinfo" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:07.944740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.944677 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-kube-api-access-nqstz" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "kube-api-access-nqstz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:07.948938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.948915 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7"] Oct 30 01:25:07.957971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.957949 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd"] Oct 30 01:25:07.985774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.985728 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-envoy" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:07.986775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.986718 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-token" pod "70491b5d-60ce-4630-997e-9f16b05bc8b4" (UID: "70491b5d-60ce-4630-997e-9f16b05bc8b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:07.988308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:07.988197 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf"] Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040414 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqstz\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-kube-api-access-nqstz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040448 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040461 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040474 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040488 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040500 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040512 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040541 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70491b5d-60ce-4630-997e-9f16b05bc8b4-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.040582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.040552 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70491b5d-60ce-4630-997e-9f16b05bc8b4-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.598673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.598641 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.657927 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b272f295-7d0a-49d2-af8d-4970059d17a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b272f295-7d0a-49d2-af8d-4970059d17a7-istiod-ca-cert\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.657996 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-envoy\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.658073 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-credential-socket\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.658118 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-podinfo\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.658291 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-token\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-token\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.662802 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-envoy" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.663048 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-credential-socket" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.663070 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b272f295-7d0a-49d2-af8d-4970059d17a7-istiod-ca-cert" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.668481 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt"] Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.674489 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-data\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.674563 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-certs\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.674620 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-kube-api-access-rvktb\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-kube-api-access-rvktb\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.674718 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-socket\") pod \"b272f295-7d0a-49d2-af8d-4970059d17a7\" (UID: \"b272f295-7d0a-49d2-af8d-4970059d17a7\") " Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.675420 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b272f295-7d0a-49d2-af8d-4970059d17a7-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.675441 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.675572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.675454 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.677017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.675678 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-socket" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:08.677017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.675844 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-data" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:08.677017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.675996 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-certs" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:08.677017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.676977 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-token" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:08.678900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.678873 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-kube-api-access-rvktb" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "kube-api-access-rvktb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:08.682916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.682778 4905 generic.go:356] "Generic (PLEG): container finished" podID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerID="6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343" exitCode=0 Oct 30 01:25:08.683769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.683591 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" event={"ID":"b272f295-7d0a-49d2-af8d-4970059d17a7","Type":"ContainerDied","Data":"6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343"} Oct 30 01:25:08.683769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.683618 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" Oct 30 01:25:08.683769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.683655 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n" event={"ID":"b272f295-7d0a-49d2-af8d-4970059d17a7","Type":"ContainerDied","Data":"1a0712635b209165648c99a6d3826d3452e0a32e1a7ccce59658c70e09de33a3"} Oct 30 01:25:08.683769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.683690 4905 scope.go:122] "RemoveContainer" containerID="6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343" Oct 30 01:25:08.687498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.687272 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-podinfo" pod "b272f295-7d0a-49d2-af8d-4970059d17a7" (UID: "b272f295-7d0a-49d2-af8d-4970059d17a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:08.694455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.694430 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" event={"ID":"70491b5d-60ce-4630-997e-9f16b05bc8b4","Type":"ContainerDied","Data":"e62109b6578f5b7c2abb2f78425ee32014fea985829afb278d41526153d16fed"} Oct 30 01:25:08.694696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.694561 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4" Oct 30 01:25:08.709991 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.709487 4905 scope.go:122] "RemoveContainer" containerID="0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25" Oct 30 01:25:08.721705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.721684 4905 scope.go:122] "RemoveContainer" containerID="6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343" Oct 30 01:25:08.722176 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:08.722042 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343\": container with ID starting with 6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343 not found: ID does not exist" containerID="6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343" Oct 30 01:25:08.722176 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.722078 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343"} err="failed to get container status \"6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343\": rpc error: code = NotFound desc = could not find container \"6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343\": container with ID starting with 6c784437ffb070f12b074eaf26e0d4b396bc15dd0da1fe623c34a336bafd4343 not found: ID does not exist" Oct 30 01:25:08.722176 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.722105 4905 scope.go:122] "RemoveContainer" containerID="0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25" Oct 30 01:25:08.722511 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:08.722484 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25\": container with ID starting with 0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25 not found: ID does not exist" containerID="0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25" Oct 30 01:25:08.722738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.722713 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25"} err="failed to get container status \"0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25\": rpc error: code = NotFound desc = could not find container \"0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25\": container with ID starting with 0c559c93054ce32c4fd2c69c2efab97e9a5abc757d032861c6dd6b1877283e25 not found: ID does not exist" Oct 30 01:25:08.722839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.722829 4905 scope.go:122] "RemoveContainer" containerID="2c87d657eeb30879083c1231814cb6a9ff9952a47827af8fdffa072bc747d921" Oct 30 01:25:08.776114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.776044 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4"] Oct 30 01:25:08.776881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.776854 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.777012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.776886 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.777012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.776900 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.777012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.776914 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvktb\" (UniqueName: \"kubernetes.io/projected/b272f295-7d0a-49d2-af8d-4970059d17a7-kube-api-access-rvktb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.777012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.776927 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b272f295-7d0a-49d2-af8d-4970059d17a7-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.777012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.776947 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b272f295-7d0a-49d2-af8d-4970059d17a7-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:08.779532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:08.779490 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-8hzg4"] Oct 30 01:25:09.046361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.045855 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n"] Oct 30 01:25:09.054357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.054315 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bdf6f3-openshift-default-5c56ff46b9-shm5n"] Oct 30 01:25:09.192135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.192100 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f264934-e2ee-435e-898f-921f20b00f16" path="/var/lib/kubelet/pods/5f264934-e2ee-435e-898f-921f20b00f16/volumes" Oct 30 01:25:09.192607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.192590 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" path="/var/lib/kubelet/pods/70491b5d-60ce-4630-997e-9f16b05bc8b4/volumes" Oct 30 01:25:09.193021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.193009 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" path="/var/lib/kubelet/pods/b272f295-7d0a-49d2-af8d-4970059d17a7/volumes" Oct 30 01:25:09.499977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.499909 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" podUID="060b152e-dbf2-491a-8d54-f3641368c588" containerName="payload-processing" containerID="cri-o://031e9201535eea0b64351acb7516a35d39a837b6daccf681c2f11257e7e60870" gracePeriod=25 Oct 30 01:25:09.548076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.548028 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" podUID="70df62fa-e007-4674-b5bc-07d10d2cf858" containerName="payload-pre-processing" containerID="cri-o://74b66c44f5fdd7482583c1d0ac32cd427298a10abc61421a59ebbe3bb5c454d7" gracePeriod=25 Oct 30 01:25:09.707393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.707356 4905 generic.go:356] "Generic (PLEG): container finished" podID="060b152e-dbf2-491a-8d54-f3641368c588" containerID="031e9201535eea0b64351acb7516a35d39a837b6daccf681c2f11257e7e60870" exitCode=0 Oct 30 01:25:09.707927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.707482 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" event={"ID":"060b152e-dbf2-491a-8d54-f3641368c588","Type":"ContainerDied","Data":"031e9201535eea0b64351acb7516a35d39a837b6daccf681c2f11257e7e60870"} Oct 30 01:25:09.709105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.709080 4905 generic.go:356] "Generic (PLEG): container finished" podID="70df62fa-e007-4674-b5bc-07d10d2cf858" containerID="74b66c44f5fdd7482583c1d0ac32cd427298a10abc61421a59ebbe3bb5c454d7" exitCode=0 Oct 30 01:25:09.709242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.709134 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" event={"ID":"70df62fa-e007-4674-b5bc-07d10d2cf858","Type":"ContainerDied","Data":"74b66c44f5fdd7482583c1d0ac32cd427298a10abc61421a59ebbe3bb5c454d7"} Oct 30 01:25:09.866306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.866276 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:25:09.904227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.904199 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:25:09.994845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.994798 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70df62fa-e007-4674-b5bc-07d10d2cf858-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70df62fa-e007-4674-b5bc-07d10d2cf858-plugins-config-volume\") pod \"70df62fa-e007-4674-b5bc-07d10d2cf858\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " Oct 30 01:25:09.995010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.994893 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70df62fa-e007-4674-b5bc-07d10d2cf858-kube-api-access-j8752\" (UniqueName: \"kubernetes.io/projected/70df62fa-e007-4674-b5bc-07d10d2cf858-kube-api-access-j8752\") pod \"70df62fa-e007-4674-b5bc-07d10d2cf858\" (UID: \"70df62fa-e007-4674-b5bc-07d10d2cf858\") " Oct 30 01:25:09.995010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.994945 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060b152e-dbf2-491a-8d54-f3641368c588-kube-api-access-vndlh\" (UniqueName: \"kubernetes.io/projected/060b152e-dbf2-491a-8d54-f3641368c588-kube-api-access-vndlh\") pod \"060b152e-dbf2-491a-8d54-f3641368c588\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " Oct 30 01:25:09.995114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.995029 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060b152e-dbf2-491a-8d54-f3641368c588-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060b152e-dbf2-491a-8d54-f3641368c588-plugins-config-volume\") pod \"060b152e-dbf2-491a-8d54-f3641368c588\" (UID: \"060b152e-dbf2-491a-8d54-f3641368c588\") " Oct 30 01:25:09.995627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.995598 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70df62fa-e007-4674-b5bc-07d10d2cf858-plugins-config-volume" pod "70df62fa-e007-4674-b5bc-07d10d2cf858" (UID: "70df62fa-e007-4674-b5bc-07d10d2cf858"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:09.995747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.995626 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060b152e-dbf2-491a-8d54-f3641368c588-plugins-config-volume" pod "060b152e-dbf2-491a-8d54-f3641368c588" (UID: "060b152e-dbf2-491a-8d54-f3641368c588"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:09.997552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.997507 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70df62fa-e007-4674-b5bc-07d10d2cf858-kube-api-access-j8752" pod "70df62fa-e007-4674-b5bc-07d10d2cf858" (UID: "70df62fa-e007-4674-b5bc-07d10d2cf858"). InnerVolumeSpecName "kube-api-access-j8752". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:09.997902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:09.997882 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060b152e-dbf2-491a-8d54-f3641368c588-kube-api-access-vndlh" pod "060b152e-dbf2-491a-8d54-f3641368c588" (UID: "060b152e-dbf2-491a-8d54-f3641368c588"). InnerVolumeSpecName "kube-api-access-vndlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:10.096225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.096143 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8752\" (UniqueName: \"kubernetes.io/projected/70df62fa-e007-4674-b5bc-07d10d2cf858-kube-api-access-j8752\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:10.096225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.096176 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndlh\" (UniqueName: \"kubernetes.io/projected/060b152e-dbf2-491a-8d54-f3641368c588-kube-api-access-vndlh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:10.096225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.096189 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060b152e-dbf2-491a-8d54-f3641368c588-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:10.096225 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.096202 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70df62fa-e007-4674-b5bc-07d10d2cf858-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:10.738261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.734879 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" Oct 30 01:25:10.738261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.734911 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d" event={"ID":"060b152e-dbf2-491a-8d54-f3641368c588","Type":"ContainerDied","Data":"9ce977dd3b4db181a81c571be0ed9534e6aa22ee02d20bfaa0ff34d07eabe192"} Oct 30 01:25:10.738261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.734980 4905 scope.go:122] "RemoveContainer" containerID="031e9201535eea0b64351acb7516a35d39a837b6daccf681c2f11257e7e60870" Oct 30 01:25:10.738261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.736888 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" event={"ID":"70df62fa-e007-4674-b5bc-07d10d2cf858","Type":"ContainerDied","Data":"956981e272ceb58922485feb6b3365aa8fa35688f459c59ce4d1de6488bcd12e"} Oct 30 01:25:10.738261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.736982 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p" Oct 30 01:25:10.751751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.751708 4905 scope.go:122] "RemoveContainer" containerID="74b66c44f5fdd7482583c1d0ac32cd427298a10abc61421a59ebbe3bb5c454d7" Oct 30 01:25:10.790977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.790952 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d"] Oct 30 01:25:10.804690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.804649 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6bd9cd6c46-5nv7d"] Oct 30 01:25:10.831430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.831349 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p"] Oct 30 01:25:10.836593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:10.836505 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-6dbf8bb7d7-44k4p"] Oct 30 01:25:11.191815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.191736 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060b152e-dbf2-491a-8d54-f3641368c588" path="/var/lib/kubelet/pods/060b152e-dbf2-491a-8d54-f3641368c588/volumes" Oct 30 01:25:11.192178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.192165 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70df62fa-e007-4674-b5bc-07d10d2cf858" path="/var/lib/kubelet/pods/70df62fa-e007-4674-b5bc-07d10d2cf858/volumes" Oct 30 01:25:11.290857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.290823 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 01:25:11.291033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.290889 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 01:25:11.749423 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.749388 4905 generic.go:356] "Generic (PLEG): container finished" podID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerID="f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd" exitCode=0 Oct 30 01:25:11.750018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.749465 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" event={"ID":"220ad5d5-3373-4eb3-a4c2-d9f068297389","Type":"ContainerDied","Data":"f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd"} Oct 30 01:25:11.750018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.749890 4905 scope.go:122] "RemoveContainer" containerID="f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd" Oct 30 01:25:11.751741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.751705 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:11.751863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.751798 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:11.751863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:11.751846 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:12.442570 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:12.442512 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:25:12.764795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:12.764755 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" event={"ID":"220ad5d5-3373-4eb3-a4c2-d9f068297389","Type":"ContainerStarted","Data":"5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75"} Oct 30 01:25:12.943544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:12.941913 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:25:12.943544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:12.942036 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 01:25:12.943544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:12.942079 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 01:25:13.663224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:13.663187 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6"] Oct 30 01:25:13.694470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:13.694436 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654"] Oct 30 01:25:13.944716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:13.938918 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 01:25:13.944716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:13.938980 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 01:25:14.945547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:14.938147 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 01:25:14.945547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:14.938206 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 01:25:15.946869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:15.946835 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 01:25:15.947482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:15.947451 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 01:25:16.944545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:16.941555 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 01:25:16.944545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:16.941668 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 01:25:17.950545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:17.946860 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 01:25:17.950545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:17.946926 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 01:25:18.664583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:18.664528 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" podUID="9569e49a-21e9-447b-b435-ff3cb63a323c" containerName="payload-processing" containerID="cri-o://f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc" gracePeriod=25 Oct 30 01:25:18.695315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:18.695260 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" podUID="10ae9f35-3087-4a5a-ab61-7c856cb32928" containerName="payload-pre-processing" containerID="cri-o://7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098" gracePeriod=25 Oct 30 01:25:18.941495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:18.941412 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:25:18.942001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:18.941974 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:25:18.942613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:18.942596 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:25:20.763967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.763931 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl"] Oct 30 01:25:20.764786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764762 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" Oct 30 01:25:20.764786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764786 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764814 4905 cpu_manager.go:414] "containerMap: removing container" podUID="060b152e-dbf2-491a-8d54-f3641368c588" containerName="payload-processing" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764822 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060b152e-dbf2-491a-8d54-f3641368c588" containerName="payload-processing" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764845 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" containerName="payload-processing" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764852 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" containerName="payload-processing" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764863 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" containerName="payload-pre-processing" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764871 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" containerName="payload-pre-processing" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764885 4905 cpu_manager.go:414] "containerMap: removing container" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764893 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764910 4905 cpu_manager.go:414] "containerMap: removing container" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764916 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764927 4905 cpu_manager.go:414] "containerMap: removing container" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764934 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764956 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" containerName="maas-api" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764963 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" containerName="maas-api" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764975 4905 cpu_manager.go:414] "containerMap: removing container" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" Oct 30 01:25:20.764976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764982 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.764996 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765003 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765030 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4931e93a-0f20-47c1-9f73-ffe763e183b1" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765037 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4931e93a-0f20-47c1-9f73-ffe763e183b1" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765049 4905 cpu_manager.go:414] "containerMap: removing container" podUID="70df62fa-e007-4674-b5bc-07d10d2cf858" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765056 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70df62fa-e007-4674-b5bc-07d10d2cf858" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765068 4905 cpu_manager.go:414] "containerMap: removing container" podUID="a469becb-c0d0-40cc-b426-3287026d49aa" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765074 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a469becb-c0d0-40cc-b426-3287026d49aa" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765099 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ede6ca67-abbc-4bda-8b14-2f3ac1eac173" containerName="maas-api" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765106 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede6ca67-abbc-4bda-8b14-2f3ac1eac173" containerName="maas-api" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765276 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765293 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="060b152e-dbf2-491a-8d54-f3641368c588" containerName="payload-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765306 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd181bda-aa31-4f73-9a62-c3485fa03d1f" containerName="payload-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765315 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7342a4a6-3d5a-4dc9-bf3b-1b8314442c7e" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765324 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765334 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765342 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="17922593-835e-4bd3-9e76-7510d501815f" containerName="istio-proxy" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765352 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2b4abf-aafe-4c3f-a52d-d05862efb7ea" containerName="maas-api" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765365 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f264934-e2ee-435e-898f-921f20b00f16" containerName="main" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765375 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4931e93a-0f20-47c1-9f73-ffe763e183b1" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765385 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="70df62fa-e007-4674-b5bc-07d10d2cf858" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765396 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="a469becb-c0d0-40cc-b426-3287026d49aa" containerName="payload-pre-processing" Oct 30 01:25:20.765869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.765407 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede6ca67-abbc-4bda-8b14-2f3ac1eac173" containerName="maas-api" Oct 30 01:25:20.769915 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.769890 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.785769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.785749 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-298ddc-openshift-default-dockercfg-lcv79\"" Oct 30 01:25:20.803706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.803679 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl"] Oct 30 01:25:20.810053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810026 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-token\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810074 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkhh\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-kube-api-access-dlkhh\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810208 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810288 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-data\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810339 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istiod-ca-cert\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810399 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-credential-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810560 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-certs\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810594 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-envoy\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.810733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.810716 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-podinfo\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911349 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911430 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-data\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911471 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istiod-ca-cert\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911553 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-credential-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911621 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-certs\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911645 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-envoy\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911714 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-podinfo\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911756 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-token\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.911821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911787 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkhh\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-kube-api-access-dlkhh\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.912087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911817 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.912087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.911909 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-data\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.912087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.912000 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-certs\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.912250 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.912214 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istiod-ca-cert\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.912407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.912384 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-credential-socket\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.914263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.914241 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-envoy\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.914460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.914444 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-podinfo\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.933079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.933044 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkhh\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-kube-api-access-dlkhh\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:20.933367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:20.933351 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-token\") pod \"e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:21.084401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.084301 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:21.401955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.401926 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl"] Oct 30 01:25:21.404074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.404028 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:21.404173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.404114 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:21.404173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.404143 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:21.842014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.841972 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" event={"ID":"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd","Type":"ContainerStarted","Data":"6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749"} Oct 30 01:25:21.842569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.842050 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" event={"ID":"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd","Type":"ContainerStarted","Data":"6a5d7b8068f6aee9cfb57e64e9eaaa92a1a451ebd7b108c922628b0259dcdb81"} Oct 30 01:25:21.905379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:21.905268 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podStartSLOduration=1.905250245 podStartE2EDuration="1.905250245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:21.899440016 +0000 UTC m=+2806.842613241" watchObservedRunningTime="2026-10-30 01:25:21.905250245 +0000 UTC m=+2806.848423438" Oct 30 01:25:22.086282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:22.086254 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:22.087785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:22.087758 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 01:25:22.087926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:22.087803 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 01:25:23.016838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.016801 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" containerID="cri-o://70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9" gracePeriod=45 Oct 30 01:25:23.021186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.021150 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" containerID="cri-o://4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d" gracePeriod=45 Oct 30 01:25:23.059113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.059074 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" containerID="cri-o://b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37" gracePeriod=45 Oct 30 01:25:23.085206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.085179 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 01:25:23.085318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.085239 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 01:25:23.289548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.288779 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Oct 30 01:25:23.471202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.471172 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:25:23.509768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.509391 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:25:23.512692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.512670 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:25:23.578564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578470 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-dshm\") pod \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " Oct 30 01:25:23.578740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578611 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9615f726-7b53-4e84-a623-49a37baa3af3-tls-certs\" (UniqueName: \"kubernetes.io/secret/9615f726-7b53-4e84-a623-49a37baa3af3-tls-certs\") pod \"9615f726-7b53-4e84-a623-49a37baa3af3\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " Oct 30 01:25:23.578740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578645 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tmp-dir\") pod \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " Oct 30 01:25:23.578740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578686 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tls-certs\" (UniqueName: \"kubernetes.io/secret/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tls-certs\") pod \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " Oct 30 01:25:23.578740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578723 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9615f726-7b53-4e84-a623-49a37baa3af3-kube-api-access-mgxdt\" (UniqueName: \"kubernetes.io/projected/9615f726-7b53-4e84-a623-49a37baa3af3-kube-api-access-mgxdt\") pod \"9615f726-7b53-4e84-a623-49a37baa3af3\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " Oct 30 01:25:23.579089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578768 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-dshm\") pod \"9615f726-7b53-4e84-a623-49a37baa3af3\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " Oct 30 01:25:23.579089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578814 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tmp-dir\") pod \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " Oct 30 01:25:23.579089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578865 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-kube-api-access-zsqfr\" (UniqueName: \"kubernetes.io/projected/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-kube-api-access-zsqfr\") pod \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " Oct 30 01:25:23.579089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578924 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-model-cache\") pod \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " Oct 30 01:25:23.579089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.578988 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-home\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-home\") pod \"9615f726-7b53-4e84-a623-49a37baa3af3\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " Oct 30 01:25:23.579089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579022 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-model-cache\") pod \"9615f726-7b53-4e84-a623-49a37baa3af3\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " Oct 30 01:25:23.579089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579055 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-kube-api-access-xx674\" (UniqueName: \"kubernetes.io/projected/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-kube-api-access-xx674\") pod \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " Oct 30 01:25:23.579468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579146 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-dshm\") pod \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " Oct 30 01:25:23.579468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579190 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-tmp-dir\") pod \"9615f726-7b53-4e84-a623-49a37baa3af3\" (UID: \"9615f726-7b53-4e84-a623-49a37baa3af3\") " Oct 30 01:25:23.579468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579268 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-home\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-home\") pod \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " Oct 30 01:25:23.579468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579326 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-home\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-home\") pod \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " Oct 30 01:25:23.579468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579409 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tls-certs\") pod \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\" (UID: \"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c\") " Oct 30 01:25:23.579468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.579447 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-model-cache\") pod \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\" (UID: \"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626\") " Oct 30 01:25:23.581838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.580749 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-model-cache" pod "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" (UID: "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.581838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.580924 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-home" pod "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" (UID: "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.581838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.580938 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-home" pod "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" (UID: "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.581838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.581300 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-tmp-dir" pod "9615f726-7b53-4e84-a623-49a37baa3af3" (UID: "9615f726-7b53-4e84-a623-49a37baa3af3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.581838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.581501 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tmp-dir" pod "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" (UID: "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.585964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.584849 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-kube-api-access-zsqfr" pod "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" (UID: "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c"). InnerVolumeSpecName "kube-api-access-zsqfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:23.585964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.585066 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tmp-dir" pod "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" (UID: "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.585964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.585285 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-model-cache" pod "9615f726-7b53-4e84-a623-49a37baa3af3" (UID: "9615f726-7b53-4e84-a623-49a37baa3af3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.585964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.585495 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-home" pod "9615f726-7b53-4e84-a623-49a37baa3af3" (UID: "9615f726-7b53-4e84-a623-49a37baa3af3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.585964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.585586 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-model-cache" pod "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" (UID: "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.588286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.588260 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9615f726-7b53-4e84-a623-49a37baa3af3-kube-api-access-mgxdt" pod "9615f726-7b53-4e84-a623-49a37baa3af3" (UID: "9615f726-7b53-4e84-a623-49a37baa3af3"). InnerVolumeSpecName "kube-api-access-mgxdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:23.590492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.590465 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-dshm" pod "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" (UID: "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.595847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.595164 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-kube-api-access-xx674" pod "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" (UID: "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626"). InnerVolumeSpecName "kube-api-access-xx674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:23.595847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.595689 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tls-certs" pod "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" (UID: "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:23.595847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.595711 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-dshm" pod "9615f726-7b53-4e84-a623-49a37baa3af3" (UID: "9615f726-7b53-4e84-a623-49a37baa3af3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.595847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.595715 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9615f726-7b53-4e84-a623-49a37baa3af3-tls-certs" pod "9615f726-7b53-4e84-a623-49a37baa3af3" (UID: "9615f726-7b53-4e84-a623-49a37baa3af3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:23.596139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.595986 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tls-certs" pod "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" (UID: "fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:23.597234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.597214 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-dshm" pod "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" (UID: "03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:23.682931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.682896 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.682931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.682930 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.682949 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx674\" (UniqueName: \"kubernetes.io/projected/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-kube-api-access-xx674\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.682962 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.682974 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.682984 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.682994 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683005 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683015 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683026 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683036 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9615f726-7b53-4e84-a623-49a37baa3af3-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683047 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683059 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683071 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxdt\" (UniqueName: \"kubernetes.io/projected/9615f726-7b53-4e84-a623-49a37baa3af3-kube-api-access-mgxdt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683081 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9615f726-7b53-4e84-a623-49a37baa3af3-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683092 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683103 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsqfr\" (UniqueName: \"kubernetes.io/projected/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-kube-api-access-zsqfr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.683181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.683115 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:23.705019 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.704979 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" containerID="cri-o://bf104aa77587f493e2a7ad1e56b408ce458e8fed3da2d787c7ffea1dce1783d8" gracePeriod=45 Oct 30 01:25:23.716753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.716713 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": dial tcp 10.132.0.93:8000: connect: connection refused" Oct 30 01:25:23.871599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.871497 4905 generic.go:356] "Generic (PLEG): container finished" podID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerID="b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37" exitCode=0 Oct 30 01:25:23.871599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.871555 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" event={"ID":"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c","Type":"ContainerDied","Data":"b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37"} Oct 30 01:25:23.871890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.871616 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" event={"ID":"03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c","Type":"ContainerDied","Data":"5b6db60c740cf70ced1f87dd0dc475d8bf2e284b136c485043fd87b6e11b846e"} Oct 30 01:25:23.871890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.871640 4905 scope.go:122] "RemoveContainer" containerID="b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37" Oct 30 01:25:23.872075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.872054 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf" Oct 30 01:25:23.874394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.874240 4905 generic.go:356] "Generic (PLEG): container finished" podID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerID="bf104aa77587f493e2a7ad1e56b408ce458e8fed3da2d787c7ffea1dce1783d8" exitCode=0 Oct 30 01:25:23.874394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.874333 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" event={"ID":"5e29d6bb-e022-4b06-ae02-5b964fc5d34e","Type":"ContainerDied","Data":"bf104aa77587f493e2a7ad1e56b408ce458e8fed3da2d787c7ffea1dce1783d8"} Oct 30 01:25:23.877293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.876342 4905 generic.go:356] "Generic (PLEG): container finished" podID="9615f726-7b53-4e84-a623-49a37baa3af3" containerID="4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d" exitCode=0 Oct 30 01:25:23.877293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.876411 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" event={"ID":"9615f726-7b53-4e84-a623-49a37baa3af3","Type":"ContainerDied","Data":"4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d"} Oct 30 01:25:23.877293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.876435 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" event={"ID":"9615f726-7b53-4e84-a623-49a37baa3af3","Type":"ContainerDied","Data":"6d41e6c1ea3ddb00800a088c2c72abbc53ae41c10f322c2b8cea51732eec97c8"} Oct 30 01:25:23.877293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.876500 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" Oct 30 01:25:23.887157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.886834 4905 generic.go:356] "Generic (PLEG): container finished" podID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerID="70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9" exitCode=0 Oct 30 01:25:23.887157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.886962 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" event={"ID":"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626","Type":"ContainerDied","Data":"70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9"} Oct 30 01:25:23.887157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.886997 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" event={"ID":"fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626","Type":"ContainerDied","Data":"4f5eea68d99dc29c20e900607246c6f66648eb3b184756517f977451be6ed9d5"} Oct 30 01:25:23.887157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.887052 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7" Oct 30 01:25:23.890891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.890657 4905 scope.go:122] "RemoveContainer" containerID="b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37" Oct 30 01:25:23.891394 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:23.891074 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37\": container with ID starting with b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37 not found: ID does not exist" containerID="b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37" Oct 30 01:25:23.891394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.891106 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37"} err="failed to get container status \"b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37\": rpc error: code = NotFound desc = could not find container \"b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37\": container with ID starting with b98256d5a74d43bc10089aeb829572bd6b3c9d13f27a9b2133927f827afbcb37 not found: ID does not exist" Oct 30 01:25:23.891394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.891127 4905 scope.go:122] "RemoveContainer" containerID="4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d" Oct 30 01:25:23.906924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.906901 4905 scope.go:122] "RemoveContainer" containerID="4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d" Oct 30 01:25:23.907265 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:23.907240 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d\": container with ID starting with 4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d not found: ID does not exist" containerID="4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d" Oct 30 01:25:23.907354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.907275 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d"} err="failed to get container status \"4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d\": rpc error: code = NotFound desc = could not find container \"4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d\": container with ID starting with 4cf52da7091efbea9d23a9eac0b56e5683a45ffe51896034922c074a8606ca3d not found: ID does not exist" Oct 30 01:25:23.907354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.907299 4905 scope.go:122] "RemoveContainer" containerID="70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9" Oct 30 01:25:23.920319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.920299 4905 scope.go:122] "RemoveContainer" containerID="70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9" Oct 30 01:25:23.920688 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:23.920667 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9\": container with ID starting with 70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9 not found: ID does not exist" containerID="70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9" Oct 30 01:25:23.920763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.920697 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9"} err="failed to get container status \"70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9\": rpc error: code = NotFound desc = could not find container \"70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9\": container with ID starting with 70a10e9f77dc424c57d16f12d34a366a1f750e0ea75ed529cd2419d87320fdc9 not found: ID does not exist" Oct 30 01:25:23.990472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.990438 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf"] Oct 30 01:25:23.995043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:23.995016 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/simulator-w6-5a344d-kserve-748fd759c4-5vwgf"] Oct 30 01:25:24.067532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.067489 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd"] Oct 30 01:25:24.089554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.089509 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 01:25:24.089788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.089762 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 01:25:24.090483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.090456 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd"] Oct 30 01:25:24.090678 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.090660 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:25:24.115688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.115663 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7"] Oct 30 01:25:24.124081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.124002 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-w6-5a344d-kserve-bd8c7b9d-fxbg7"] Oct 30 01:25:24.191259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.191222 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tls-certs\") pod \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " Oct 30 01:25:24.191452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.191429 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-home\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-home\") pod \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " Oct 30 01:25:24.191541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.191464 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-kube-api-access-bh97r\" (UniqueName: \"kubernetes.io/projected/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-kube-api-access-bh97r\") pod \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " Oct 30 01:25:24.191601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.191539 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tmp-dir\") pod \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " Oct 30 01:25:24.191601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.191572 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-dshm\") pod \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " Oct 30 01:25:24.191713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.191626 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-model-cache\") pod \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\" (UID: \"5e29d6bb-e022-4b06-ae02-5b964fc5d34e\") " Oct 30 01:25:24.192700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.192654 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-home" pod "5e29d6bb-e022-4b06-ae02-5b964fc5d34e" (UID: "5e29d6bb-e022-4b06-ae02-5b964fc5d34e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:24.192700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.192669 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-model-cache" pod "5e29d6bb-e022-4b06-ae02-5b964fc5d34e" (UID: "5e29d6bb-e022-4b06-ae02-5b964fc5d34e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:24.193903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.193849 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tmp-dir" pod "5e29d6bb-e022-4b06-ae02-5b964fc5d34e" (UID: "5e29d6bb-e022-4b06-ae02-5b964fc5d34e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:24.195956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.195934 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-dshm" pod "5e29d6bb-e022-4b06-ae02-5b964fc5d34e" (UID: "5e29d6bb-e022-4b06-ae02-5b964fc5d34e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:24.198953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.198925 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tls-certs" pod "5e29d6bb-e022-4b06-ae02-5b964fc5d34e" (UID: "5e29d6bb-e022-4b06-ae02-5b964fc5d34e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:24.199075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.199055 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-kube-api-access-bh97r" pod "5e29d6bb-e022-4b06-ae02-5b964fc5d34e" (UID: "5e29d6bb-e022-4b06-ae02-5b964fc5d34e"). InnerVolumeSpecName "kube-api-access-bh97r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:24.293370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.293338 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.293370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.293368 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.293620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.293378 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh97r\" (UniqueName: \"kubernetes.io/projected/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-kube-api-access-bh97r\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.293620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.293389 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.293620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.293399 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.293620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.293406 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29d6bb-e022-4b06-ae02-5b964fc5d34e-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.350544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.335607 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/premium-w6-5a344d-kserve-69cb799899-xx9fd" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:25:24.897988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.897955 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" Oct 30 01:25:24.898158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.897985 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt" event={"ID":"5e29d6bb-e022-4b06-ae02-5b964fc5d34e","Type":"ContainerDied","Data":"945a08f8d4abf3943499c19bd342202bd4f2f78ed7c6f89a1553775f32979e4b"} Oct 30 01:25:24.898158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.898037 4905 scope.go:122] "RemoveContainer" containerID="bf104aa77587f493e2a7ad1e56b408ce458e8fed3da2d787c7ffea1dce1783d8" Oct 30 01:25:24.969485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.969457 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt"] Oct 30 01:25:24.983891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:24.983857 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-2-w5-da76a2-kserve-86bc5f7678-g68pt"] Oct 30 01:25:25.088463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.087767 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 01:25:25.088463 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.087831 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 01:25:25.191304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.191210 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" path="/var/lib/kubelet/pods/03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c/volumes" Oct 30 01:25:25.191727 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.191710 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" path="/var/lib/kubelet/pods/5e29d6bb-e022-4b06-ae02-5b964fc5d34e/volumes" Oct 30 01:25:25.192130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.192114 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" path="/var/lib/kubelet/pods/9615f726-7b53-4e84-a623-49a37baa3af3/volumes" Oct 30 01:25:25.192678 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.192655 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" path="/var/lib/kubelet/pods/fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626/volumes" Oct 30 01:25:25.851219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.851157 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" podUID="10ae9f35-3087-4a5a-ab61-7c856cb32928" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:25.851418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:25.851272 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" podUID="9569e49a-21e9-447b-b435-ff3cb63a323c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:26.103067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.102977 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 01:25:26.103685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.103647 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 01:25:26.395833 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.395745 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9"] Oct 30 01:25:26.397037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397009 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" Oct 30 01:25:26.397189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397177 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" Oct 30 01:25:26.397269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397261 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" Oct 30 01:25:26.397352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397343 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" Oct 30 01:25:26.397431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397423 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" Oct 30 01:25:26.397502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397493 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" Oct 30 01:25:26.397591 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397582 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" Oct 30 01:25:26.397653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397645 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" Oct 30 01:25:26.397733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397724 4905 cpu_manager.go:414] "containerMap: removing container" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" Oct 30 01:25:26.397793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.397785 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" Oct 30 01:25:26.398031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.398019 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b272f295-7d0a-49d2-af8d-4970059d17a7" containerName="istio-proxy" Oct 30 01:25:26.398113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.398104 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafb0e7a-6aaf-49a6-9ab5-c7f11d32c626" containerName="main" Oct 30 01:25:26.398179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.398172 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9615f726-7b53-4e84-a623-49a37baa3af3" containerName="main" Oct 30 01:25:26.398252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.398243 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e29d6bb-e022-4b06-ae02-5b964fc5d34e" containerName="main" Oct 30 01:25:26.398312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.398304 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cbfb3c-86d3-4d4c-bf04-320b5be8ae1c" containerName="main" Oct 30 01:25:26.398394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.398385 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="70491b5d-60ce-4630-997e-9f16b05bc8b4" containerName="istio-proxy" Oct 30 01:25:26.446076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.446043 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.469121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.467833 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9"] Oct 30 01:25:26.481934 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.481791 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0519f6\"" Oct 30 01:25:26.481934 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.481892 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0519f6-dockercfg-k454n\"" Oct 30 01:25:26.558562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.558092 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.558562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.558300 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tckx\" (UniqueName: \"kubernetes.io/projected/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-kube-api-access-9tckx\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.566751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.566713 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g"] Oct 30 01:25:26.571489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.571444 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:26.592290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.591760 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g"] Oct 30 01:25:26.664055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.663941 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79trm\" (UniqueName: \"kubernetes.io/projected/7e1f673b-e586-44f2-acc4-d67c2fe94bef-kube-api-access-79trm\") pod \"payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:26.664055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.664002 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tckx\" (UniqueName: \"kubernetes.io/projected/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-kube-api-access-9tckx\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.664453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.664101 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1f673b-e586-44f2-acc4-d67c2fe94bef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:26.664453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.664158 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.665456 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.665163 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.686177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.686147 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tckx\" (UniqueName: \"kubernetes.io/projected/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-kube-api-access-9tckx\") pod \"payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.766506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.766467 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79trm\" (UniqueName: \"kubernetes.io/projected/7e1f673b-e586-44f2-acc4-d67c2fe94bef-kube-api-access-79trm\") pod \"payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:26.766698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.766593 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1f673b-e586-44f2-acc4-d67c2fe94bef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:26.767311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.767286 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1f673b-e586-44f2-acc4-d67c2fe94bef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:26.787071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.787044 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79trm\" (UniqueName: \"kubernetes.io/projected/7e1f673b-e586-44f2-acc4-d67c2fe94bef-kube-api-access-79trm\") pod \"payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:26.806695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.806672 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:26.889073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:26.888024 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:27.085994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.085582 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 01:25:27.085994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.085635 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 01:25:27.167677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.167316 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9"] Oct 30 01:25:27.258758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.258732 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g"] Oct 30 01:25:27.260838 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:25:27.260808 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1f673b_e586_44f2_acc4_d67c2fe94bef.slice/crio-f5286467daa176883768a01f8378a1b5d431c959820e2615e3e174e106237657 WatchSource:0}: Error finding container f5286467daa176883768a01f8378a1b5d431c959820e2615e3e174e106237657: Status 404 returned error can't find the container with id f5286467daa176883768a01f8378a1b5d431c959820e2615e3e174e106237657 Oct 30 01:25:27.938900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.938852 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" event={"ID":"7e1f673b-e586-44f2-acc4-d67c2fe94bef","Type":"ContainerStarted","Data":"76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e"} Oct 30 01:25:27.939110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.938950 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" event={"ID":"7e1f673b-e586-44f2-acc4-d67c2fe94bef","Type":"ContainerStarted","Data":"f5286467daa176883768a01f8378a1b5d431c959820e2615e3e174e106237657"} Oct 30 01:25:27.939110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.939074 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:27.941070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.941038 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" event={"ID":"736ddaa0-2e2d-42c8-8277-0f9ffe3da974","Type":"ContainerStarted","Data":"0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d"} Oct 30 01:25:27.941226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.941091 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" event={"ID":"736ddaa0-2e2d-42c8-8277-0f9ffe3da974","Type":"ContainerStarted","Data":"b1b5c04b99c83f582dc4cb190c71fee55962cd4f7e1f07f61144825340878e67"} Oct 30 01:25:27.941226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.941120 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:27.984259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:27.984200 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" podStartSLOduration=1.984182186 podStartE2EDuration="1.984182186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:27.980859251 +0000 UTC m=+2812.924032445" watchObservedRunningTime="2026-10-30 01:25:27.984182186 +0000 UTC m=+2812.927355383" Oct 30 01:25:28.763737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.763674 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" podStartSLOduration=2.763654944 podStartE2EDuration="2.763654944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:28.029263444 +0000 UTC m=+2812.972436638" watchObservedRunningTime="2026-10-30 01:25:28.763654944 +0000 UTC m=+2813.706828142" Oct 30 01:25:28.768368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.768339 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8"] Oct 30 01:25:28.773509 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.773485 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:28.795404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.795380 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8"] Oct 30 01:25:28.808542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.803908 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:25:28.808542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.803935 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:25:28.926968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.926933 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcdh\" (UniqueName: \"kubernetes.io/projected/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-kube-api-access-4lcdh\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:28.927328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.927305 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:28.927494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:28.927479 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.028413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.028317 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.031978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.031950 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcdh\" (UniqueName: \"kubernetes.io/projected/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-kube-api-access-4lcdh\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.032788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.030241 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.037892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.037003 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.050587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.049424 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.056107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.056027 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcdh\" (UniqueName: \"kubernetes.io/projected/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-kube-api-access-4lcdh\") pod \"maas-api-revoke-sub-1145baaf-qggg8\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.099011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.098983 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:29.126720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.126687 4905 patch_prober.go:29] interesting pod/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:25:29.126865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.126748 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:25:29.458479 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.458444 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8"] Oct 30 01:25:29.893030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.892188 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl"] Oct 30 01:25:29.893030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.892441 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" containerID="cri-o://6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749" gracePeriod=30 Oct 30 01:25:29.977780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:29.974897 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" event={"ID":"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc","Type":"ContainerStarted","Data":"efe2d74fe1ea987ec64af39de48e13624083c848486d3bc423481505502590e8"} Oct 30 01:25:30.090753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:30.090625 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:30.987959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:30.987923 4905 generic.go:356] "Generic (PLEG): container finished" podID="8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" containerID="1c10b19c7754c9b43c9c2af6e01cad5ea51de7c194279456291810bfe8d46cbf" exitCode=0 Oct 30 01:25:30.988410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:30.988012 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" event={"ID":"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc","Type":"ContainerDied","Data":"1c10b19c7754c9b43c9c2af6e01cad5ea51de7c194279456291810bfe8d46cbf"} Oct 30 01:25:32.167415 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.167388 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:32.261699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.261593 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-xks-ca\") pod \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " Oct 30 01:25:32.261925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.261760 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-openshift-ca\") pod \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " Oct 30 01:25:32.261925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.261806 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-kube-api-access-4lcdh\" (UniqueName: \"kubernetes.io/projected/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-kube-api-access-4lcdh\") pod \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\" (UID: \"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc\") " Oct 30 01:25:32.262705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.262676 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-openshift-ca" pod "8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" (UID: "8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:32.265533 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.265458 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-kube-api-access-4lcdh" pod "8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" (UID: "8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc"). InnerVolumeSpecName "kube-api-access-4lcdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:32.266239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.266210 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-xks-ca" pod "8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" (UID: "8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:32.363498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.363460 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:32.363498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.363494 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:32.363498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:32.363506 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lcdh\" (UniqueName: \"kubernetes.io/projected/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc-kube-api-access-4lcdh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:33.004143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:33.004113 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" Oct 30 01:25:33.004317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:33.004117 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8" event={"ID":"8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc","Type":"ContainerDied","Data":"efe2d74fe1ea987ec64af39de48e13624083c848486d3bc423481505502590e8"} Oct 30 01:25:33.004317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:33.004249 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efe2d74fe1ea987ec64af39de48e13624083c848486d3bc423481505502590e8" Oct 30 01:25:33.891547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:33.891497 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p"] Oct 30 01:25:33.892013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:33.891723 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" podUID="58aad8a0-1157-439a-84f7-7ca03ab693b1" containerName="payload-processing" containerID="cri-o://51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568" gracePeriod=30 Oct 30 01:25:33.908607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:33.908577 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv"] Oct 30 01:25:33.908863 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:33.908815 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" podUID="63f76eb2-bf0e-4ffe-8653-d5596d1b2037" containerName="payload-pre-processing" containerID="cri-o://705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3" gracePeriod=30 Oct 30 01:25:34.361628 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.350980 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:25:34.415169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.415141 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:25:34.545749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.545715 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-plugins-config-volume\") pod \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " Oct 30 01:25:34.545952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.545838 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-kube-api-access-m2zfg\" (UniqueName: \"kubernetes.io/projected/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-kube-api-access-m2zfg\") pod \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\" (UID: \"63f76eb2-bf0e-4ffe-8653-d5596d1b2037\") " Oct 30 01:25:34.545952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.545877 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aad8a0-1157-439a-84f7-7ca03ab693b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aad8a0-1157-439a-84f7-7ca03ab693b1-plugins-config-volume\") pod \"58aad8a0-1157-439a-84f7-7ca03ab693b1\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " Oct 30 01:25:34.545952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.545929 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aad8a0-1157-439a-84f7-7ca03ab693b1-kube-api-access-cdbs2\" (UniqueName: \"kubernetes.io/projected/58aad8a0-1157-439a-84f7-7ca03ab693b1-kube-api-access-cdbs2\") pod \"58aad8a0-1157-439a-84f7-7ca03ab693b1\" (UID: \"58aad8a0-1157-439a-84f7-7ca03ab693b1\") " Oct 30 01:25:34.546231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.546203 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-plugins-config-volume" pod "63f76eb2-bf0e-4ffe-8653-d5596d1b2037" (UID: "63f76eb2-bf0e-4ffe-8653-d5596d1b2037"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:34.546695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.546665 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aad8a0-1157-439a-84f7-7ca03ab693b1-plugins-config-volume" pod "58aad8a0-1157-439a-84f7-7ca03ab693b1" (UID: "58aad8a0-1157-439a-84f7-7ca03ab693b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:34.546916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.546888 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:34.546988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.546922 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aad8a0-1157-439a-84f7-7ca03ab693b1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:34.548753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.548725 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-kube-api-access-m2zfg" pod "63f76eb2-bf0e-4ffe-8653-d5596d1b2037" (UID: "63f76eb2-bf0e-4ffe-8653-d5596d1b2037"). InnerVolumeSpecName "kube-api-access-m2zfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:34.549135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.549114 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aad8a0-1157-439a-84f7-7ca03ab693b1-kube-api-access-cdbs2" pod "58aad8a0-1157-439a-84f7-7ca03ab693b1" (UID: "58aad8a0-1157-439a-84f7-7ca03ab693b1"). InnerVolumeSpecName "kube-api-access-cdbs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:34.648703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.648652 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2zfg\" (UniqueName: \"kubernetes.io/projected/63f76eb2-bf0e-4ffe-8653-d5596d1b2037-kube-api-access-m2zfg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:34.648703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:34.648698 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdbs2\" (UniqueName: \"kubernetes.io/projected/58aad8a0-1157-439a-84f7-7ca03ab693b1-kube-api-access-cdbs2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.020354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.020319 4905 generic.go:356] "Generic (PLEG): container finished" podID="58aad8a0-1157-439a-84f7-7ca03ab693b1" containerID="51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568" exitCode=0 Oct 30 01:25:35.020871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.020365 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" Oct 30 01:25:35.020871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.020414 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" event={"ID":"58aad8a0-1157-439a-84f7-7ca03ab693b1","Type":"ContainerDied","Data":"51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568"} Oct 30 01:25:35.020871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.020468 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p" event={"ID":"58aad8a0-1157-439a-84f7-7ca03ab693b1","Type":"ContainerDied","Data":"b7f032db2844d7d3a50d25379cda72b23c2d75506c942a5a2552aeb9064a8354"} Oct 30 01:25:35.020871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.020495 4905 scope.go:122] "RemoveContainer" containerID="51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568" Oct 30 01:25:35.022108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.022085 4905 generic.go:356] "Generic (PLEG): container finished" podID="63f76eb2-bf0e-4ffe-8653-d5596d1b2037" containerID="705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3" exitCode=0 Oct 30 01:25:35.022239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.022132 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" Oct 30 01:25:35.022239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.022207 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" event={"ID":"63f76eb2-bf0e-4ffe-8653-d5596d1b2037","Type":"ContainerDied","Data":"705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3"} Oct 30 01:25:35.022239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.022238 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv" event={"ID":"63f76eb2-bf0e-4ffe-8653-d5596d1b2037","Type":"ContainerDied","Data":"0197322f51d41682e699e9f1dbf47b5aca86a2c714bc53b3c07fd40dd3df8a51"} Oct 30 01:25:35.036223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.036178 4905 scope.go:122] "RemoveContainer" containerID="51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568" Oct 30 01:25:35.036587 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:35.036564 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568\": container with ID starting with 51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568 not found: ID does not exist" containerID="51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568" Oct 30 01:25:35.036683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.036595 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568"} err="failed to get container status \"51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568\": rpc error: code = NotFound desc = could not find container \"51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568\": container with ID starting with 51cd83f23c2ab1b459787524eb226e8a1c1f714b1f196b3e89833252d40e4568 not found: ID does not exist" Oct 30 01:25:35.036683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.036613 4905 scope.go:122] "RemoveContainer" containerID="705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3" Oct 30 01:25:35.048233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.048209 4905 scope.go:122] "RemoveContainer" containerID="705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3" Oct 30 01:25:35.048580 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:35.048556 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3\": container with ID starting with 705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3 not found: ID does not exist" containerID="705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3" Oct 30 01:25:35.048668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.048591 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3"} err="failed to get container status \"705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3\": rpc error: code = NotFound desc = could not find container \"705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3\": container with ID starting with 705a53a53d1cced029c7302d6ea168127c5e20a4bbf0d42e30adfaa2bf1c0db3 not found: ID does not exist" Oct 30 01:25:35.078042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.078010 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p"] Oct 30 01:25:35.082208 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.081896 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-8c6c8f5d8-rgc9p"] Oct 30 01:25:35.121324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.121285 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv"] Oct 30 01:25:35.126402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.126372 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-754cfc6455-rcpdv"] Oct 30 01:25:35.192434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.192402 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aad8a0-1157-439a-84f7-7ca03ab693b1" path="/var/lib/kubelet/pods/58aad8a0-1157-439a-84f7-7ca03ab693b1/volumes" Oct 30 01:25:35.193121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.193098 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f76eb2-bf0e-4ffe-8653-d5596d1b2037" path="/var/lib/kubelet/pods/63f76eb2-bf0e-4ffe-8653-d5596d1b2037/volumes" Oct 30 01:25:35.421852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.421791 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:35.458402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458366 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-kube-api-access-dlkhh\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-kube-api-access-dlkhh\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458455 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-credential-socket\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458491 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-token\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-token\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458546 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-data\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458589 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-podinfo\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458673 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-envoy\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458700 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-certs\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458740 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istiod-ca-cert\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458771 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-socket\") pod \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\" (UID: \"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd\") " Oct 30 01:25:35.458841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458773 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-credential-socket" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:35.459106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.458923 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-data" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:35.460061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.459138 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-socket" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:35.460061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.459312 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-certs" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:35.460061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.459618 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.460061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.459670 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.460061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.459686 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.460061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.459701 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.460061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.460019 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istiod-ca-cert" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:35.461252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.461223 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-envoy" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:35.462025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.462000 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-kube-api-access-dlkhh" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "kube-api-access-dlkhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:35.462258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.462232 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-podinfo" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:35.462744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.462697 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-token" pod "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" (UID: "75d6262c-07f2-42b4-94dd-c5d41ba5c0dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:35.561155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.561072 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkhh\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-kube-api-access-dlkhh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.561155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.561149 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.561155 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.561159 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.561486 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.561167 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.561486 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.561177 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:35.850070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.850025 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" podUID="9569e49a-21e9-447b-b435-ff3cb63a323c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:35.850070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:35.850040 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" podUID="10ae9f35-3087-4a5a-ab61-7c856cb32928" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:36.037595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.037558 4905 generic.go:356] "Generic (PLEG): container finished" podID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerID="6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749" exitCode=0 Oct 30 01:25:36.038087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.037654 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" event={"ID":"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd","Type":"ContainerDied","Data":"6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749"} Oct 30 01:25:36.038087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.037706 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" event={"ID":"75d6262c-07f2-42b4-94dd-c5d41ba5c0dd","Type":"ContainerDied","Data":"6a5d7b8068f6aee9cfb57e64e9eaaa92a1a451ebd7b108c922628b0259dcdb81"} Oct 30 01:25:36.038087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.037736 4905 scope.go:122] "RemoveContainer" containerID="6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749" Oct 30 01:25:36.038087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.037890 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl" Oct 30 01:25:36.053396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.053372 4905 scope.go:122] "RemoveContainer" containerID="6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749" Oct 30 01:25:36.053750 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:36.053723 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749\": container with ID starting with 6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749 not found: ID does not exist" containerID="6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749" Oct 30 01:25:36.053876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.053761 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749"} err="failed to get container status \"6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749\": rpc error: code = NotFound desc = could not find container \"6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749\": container with ID starting with 6d5f502aa06af3793b7dc6e483981b7b3e8bddbdc4ec5355c4d2237b0ad83749 not found: ID does not exist" Oct 30 01:25:36.095393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.095364 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl"] Oct 30 01:25:36.109784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:36.109705 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-298ddc-openshift-default-64b9897bb7-fhwkl"] Oct 30 01:25:37.191276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:37.191234 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" path="/var/lib/kubelet/pods/75d6262c-07f2-42b4-94dd-c5d41ba5c0dd/volumes" Oct 30 01:25:38.963094 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:38.962201 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:25:38.968347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:38.968312 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:25:38.989705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:38.989670 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 01:25:39.039438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.039416 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 01:25:39.065896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.065823 4905 generic.go:356] "Generic (PLEG): container finished" podID="10ae9f35-3087-4a5a-ab61-7c856cb32928" containerID="7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098" exitCode=0 Oct 30 01:25:39.066166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.066062 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" event={"ID":"10ae9f35-3087-4a5a-ab61-7c856cb32928","Type":"ContainerDied","Data":"7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098"} Oct 30 01:25:39.066166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.066151 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" event={"ID":"10ae9f35-3087-4a5a-ab61-7c856cb32928","Type":"ContainerDied","Data":"040513bca08d85c8416e9af7f3c684d0fc46e8263aded44e49ac178a83a6ace6"} Oct 30 01:25:39.066322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.066185 4905 scope.go:122] "RemoveContainer" containerID="7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098" Oct 30 01:25:39.066399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.066333 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654" Oct 30 01:25:39.069427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.069399 4905 generic.go:356] "Generic (PLEG): container finished" podID="9569e49a-21e9-447b-b435-ff3cb63a323c" containerID="f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc" exitCode=0 Oct 30 01:25:39.069675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.069438 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" event={"ID":"9569e49a-21e9-447b-b435-ff3cb63a323c","Type":"ContainerDied","Data":"f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc"} Oct 30 01:25:39.069675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.069476 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" event={"ID":"9569e49a-21e9-447b-b435-ff3cb63a323c","Type":"ContainerDied","Data":"3c67f060e2e62c527d3c1552d2695e0eff6cd5abf97f548f2a15f39c859be95c"} Oct 30 01:25:39.069675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.069603 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6" Oct 30 01:25:39.080372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.080351 4905 scope.go:122] "RemoveContainer" containerID="7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098" Oct 30 01:25:39.080739 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:39.080714 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098\": container with ID starting with 7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098 not found: ID does not exist" containerID="7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098" Oct 30 01:25:39.080854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.080748 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098"} err="failed to get container status \"7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098\": rpc error: code = NotFound desc = could not find container \"7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098\": container with ID starting with 7404f09721e3a7c012b19c4899debfe8f14c0260f48d2c23b623159754d1a098 not found: ID does not exist" Oct 30 01:25:39.080854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.080771 4905 scope.go:122] "RemoveContainer" containerID="f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc" Oct 30 01:25:39.095499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.095455 4905 scope.go:122] "RemoveContainer" containerID="f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc" Oct 30 01:25:39.096291 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:39.096107 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc\": container with ID starting with f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc not found: ID does not exist" containerID="f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc" Oct 30 01:25:39.096941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.096486 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc"} err="failed to get container status \"f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc\": rpc error: code = NotFound desc = could not find container \"f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc\": container with ID starting with f699900dbf7020ccf5d593d00098f20d6384635d73d69a12202a0867fd534bcc not found: ID does not exist" Oct 30 01:25:39.117163 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.117133 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9569e49a-21e9-447b-b435-ff3cb63a323c-kube-api-access-rf2qt\" (UniqueName: \"kubernetes.io/projected/9569e49a-21e9-447b-b435-ff3cb63a323c-kube-api-access-rf2qt\") pod \"9569e49a-21e9-447b-b435-ff3cb63a323c\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " Oct 30 01:25:39.117324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.117235 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9569e49a-21e9-447b-b435-ff3cb63a323c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9569e49a-21e9-447b-b435-ff3cb63a323c-plugins-config-volume\") pod \"9569e49a-21e9-447b-b435-ff3cb63a323c\" (UID: \"9569e49a-21e9-447b-b435-ff3cb63a323c\") " Oct 30 01:25:39.117408 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.117349 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ae9f35-3087-4a5a-ab61-7c856cb32928-kube-api-access-r526b\" (UniqueName: \"kubernetes.io/projected/10ae9f35-3087-4a5a-ab61-7c856cb32928-kube-api-access-r526b\") pod \"10ae9f35-3087-4a5a-ab61-7c856cb32928\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " Oct 30 01:25:39.117474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.117417 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ae9f35-3087-4a5a-ab61-7c856cb32928-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ae9f35-3087-4a5a-ab61-7c856cb32928-plugins-config-volume\") pod \"10ae9f35-3087-4a5a-ab61-7c856cb32928\" (UID: \"10ae9f35-3087-4a5a-ab61-7c856cb32928\") " Oct 30 01:25:39.117993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.117966 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9569e49a-21e9-447b-b435-ff3cb63a323c-plugins-config-volume" pod "9569e49a-21e9-447b-b435-ff3cb63a323c" (UID: "9569e49a-21e9-447b-b435-ff3cb63a323c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:39.119294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.119265 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ae9f35-3087-4a5a-ab61-7c856cb32928-plugins-config-volume" pod "10ae9f35-3087-4a5a-ab61-7c856cb32928" (UID: "10ae9f35-3087-4a5a-ab61-7c856cb32928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:39.120325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.120299 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9569e49a-21e9-447b-b435-ff3cb63a323c-kube-api-access-rf2qt" pod "9569e49a-21e9-447b-b435-ff3cb63a323c" (UID: "9569e49a-21e9-447b-b435-ff3cb63a323c"). InnerVolumeSpecName "kube-api-access-rf2qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:39.121811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.121785 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ae9f35-3087-4a5a-ab61-7c856cb32928-kube-api-access-r526b" pod "10ae9f35-3087-4a5a-ab61-7c856cb32928" (UID: "10ae9f35-3087-4a5a-ab61-7c856cb32928"). InnerVolumeSpecName "kube-api-access-r526b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:39.219165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.219136 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r526b\" (UniqueName: \"kubernetes.io/projected/10ae9f35-3087-4a5a-ab61-7c856cb32928-kube-api-access-r526b\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.219280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.219169 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ae9f35-3087-4a5a-ab61-7c856cb32928-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.219280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.219185 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2qt\" (UniqueName: \"kubernetes.io/projected/9569e49a-21e9-447b-b435-ff3cb63a323c-kube-api-access-rf2qt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.219280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.219199 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9569e49a-21e9-447b-b435-ff3cb63a323c-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.408716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.408681 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654"] Oct 30 01:25:39.412638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.412613 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a344d-6759b457f5-fm654"] Oct 30 01:25:39.448885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.448834 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6"] Oct 30 01:25:39.453757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:39.453729 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a344d-79df9865cb-xqkh6"] Oct 30 01:25:41.192215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.192170 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ae9f35-3087-4a5a-ab61-7c856cb32928" path="/var/lib/kubelet/pods/10ae9f35-3087-4a5a-ab61-7c856cb32928/volumes" Oct 30 01:25:41.192813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.192792 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9569e49a-21e9-447b-b435-ff3cb63a323c" path="/var/lib/kubelet/pods/9569e49a-21e9-447b-b435-ff3cb63a323c/volumes" Oct 30 01:25:41.491985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.491938 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b"] Oct 30 01:25:41.492904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.492878 4905 cpu_manager.go:414] "containerMap: removing container" podUID="58aad8a0-1157-439a-84f7-7ca03ab693b1" containerName="payload-processing" Oct 30 01:25:41.492904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.492903 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aad8a0-1157-439a-84f7-7ca03ab693b1" containerName="payload-processing" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.492932 4905 cpu_manager.go:414] "containerMap: removing container" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.492940 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.492962 4905 cpu_manager.go:414] "containerMap: removing container" podUID="10ae9f35-3087-4a5a-ab61-7c856cb32928" containerName="payload-pre-processing" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.492970 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ae9f35-3087-4a5a-ab61-7c856cb32928" containerName="payload-pre-processing" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.492992 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9569e49a-21e9-447b-b435-ff3cb63a323c" containerName="payload-processing" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493000 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9569e49a-21e9-447b-b435-ff3cb63a323c" containerName="payload-processing" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493033 4905 cpu_manager.go:414] "containerMap: removing container" podUID="63f76eb2-bf0e-4ffe-8653-d5596d1b2037" containerName="payload-pre-processing" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493041 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f76eb2-bf0e-4ffe-8653-d5596d1b2037" containerName="payload-pre-processing" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493072 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" containerName="revoke-keys" Oct 30 01:25:41.493099 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493078 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" containerName="revoke-keys" Oct 30 01:25:41.493617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493217 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d6262c-07f2-42b4-94dd-c5d41ba5c0dd" containerName="istio-proxy" Oct 30 01:25:41.493617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493246 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ae9f35-3087-4a5a-ab61-7c856cb32928" containerName="payload-pre-processing" Oct 30 01:25:41.493617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493271 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9569e49a-21e9-447b-b435-ff3cb63a323c" containerName="payload-processing" Oct 30 01:25:41.493617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493283 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f76eb2-bf0e-4ffe-8653-d5596d1b2037" containerName="payload-pre-processing" Oct 30 01:25:41.493617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493295 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" containerName="revoke-keys" Oct 30 01:25:41.493617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.493304 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aad8a0-1157-439a-84f7-7ca03ab693b1" containerName="payload-processing" Oct 30 01:25:41.498752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.498727 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.522815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.522788 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6c37c9-openshift-default-dockercfg-79nnx\"" Oct 30 01:25:41.552528 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.552479 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b"] Oct 30 01:25:41.576088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576057 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-credential-socket\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576384 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-certs\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576433 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-socket\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576475 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-envoy\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576571 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vh4\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-kube-api-access-v6vh4\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576639 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72332109-52dc-4c45-9507-a45b122b54d0-istiod-ca-cert\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576692 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-istio-token\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576751 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72332109-52dc-4c45-9507-a45b122b54d0-istio-podinfo\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.576919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.576803 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-data\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.583871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.583838 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd"] Oct 30 01:25:41.584114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.584087 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" containerID="cri-o://46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b" gracePeriod=30 Oct 30 01:25:41.678185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678154 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-certs\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678193 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-socket\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678233 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-envoy\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678252 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vh4\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-kube-api-access-v6vh4\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678289 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72332109-52dc-4c45-9507-a45b122b54d0-istiod-ca-cert\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678323 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-istio-token\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678391 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72332109-52dc-4c45-9507-a45b122b54d0-istio-podinfo\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678426 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-data\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678554 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-credential-socket\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678675 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-certs\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.678832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.678774 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-socket\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.679233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.679210 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72332109-52dc-4c45-9507-a45b122b54d0-istiod-ca-cert\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.679233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.679224 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-credential-socket\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.679383 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.679247 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-data\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.681216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.681192 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-envoy\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.681301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.681264 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72332109-52dc-4c45-9507-a45b122b54d0-istio-podinfo\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.722238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.722170 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-istio-token\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.723370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.723342 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vh4\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-kube-api-access-v6vh4\") pod \"e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:41.823222 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:41.822814 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:42.125226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:42.125142 4905 patch_prober.go:29] interesting pod/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:25:42.125368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:42.125216 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:25:42.174489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:42.174455 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b"] Oct 30 01:25:42.174895 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:25:42.174869 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72332109_52dc_4c45_9507_a45b122b54d0.slice/crio-422c9c542782d39050a18e8e1bc4ad07f3d5eb8d182465e3d8a811d1a2eddc8e WatchSource:0}: Error finding container 422c9c542782d39050a18e8e1bc4ad07f3d5eb8d182465e3d8a811d1a2eddc8e: Status 404 returned error can't find the container with id 422c9c542782d39050a18e8e1bc4ad07f3d5eb8d182465e3d8a811d1a2eddc8e Oct 30 01:25:42.177543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:42.177493 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:42.177632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:42.177576 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:42.177632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:42.177624 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:25:42.416762 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:42.416672 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:25:43.111689 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:43.111654 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" event={"ID":"72332109-52dc-4c45-9507-a45b122b54d0","Type":"ContainerStarted","Data":"2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d"} Oct 30 01:25:43.111874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:43.111714 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" event={"ID":"72332109-52dc-4c45-9507-a45b122b54d0","Type":"ContainerStarted","Data":"422c9c542782d39050a18e8e1bc4ad07f3d5eb8d182465e3d8a811d1a2eddc8e"} Oct 30 01:25:43.160503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:43.160437 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" podStartSLOduration=2.160417092 podStartE2EDuration="2.160417092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:43.156598939 +0000 UTC m=+2828.099772128" watchObservedRunningTime="2026-10-30 01:25:43.160417092 +0000 UTC m=+2828.103590289" Oct 30 01:25:43.824813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:43.824769 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:43.830848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:43.830822 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:44.120264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:44.120189 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:44.121121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:44.121103 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:25:46.948780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:46.948758 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:25:47.062619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.062575 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-socket\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.062801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.062638 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-token\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-token\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.062801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.062672 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-kube-api-access-nvwqp\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-kube-api-access-nvwqp\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.062801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.062700 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-podinfo\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.062801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.062745 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-envoy\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.063215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.062878 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-credential-socket\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.063215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.062986 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-certs\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.063215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.063040 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istiod-ca-cert\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.063215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.063170 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-data\") pod \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\" (UID: \"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d\") " Oct 30 01:25:47.064143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.064116 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-data" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:47.065154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.065126 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-certs" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:47.065376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.065352 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-credential-socket" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:47.065543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.065388 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istiod-ca-cert" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:47.065690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.065665 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-socket" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:47.073667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.073635 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-podinfo" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:47.073806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.073722 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-token" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:47.074010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.073987 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-envoy" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:47.075047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.075023 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-kube-api-access-nvwqp" pod "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" (UID: "fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d"). InnerVolumeSpecName "kube-api-access-nvwqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:47.145129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.145018 4905 generic.go:356] "Generic (PLEG): container finished" podID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerID="46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b" exitCode=0 Oct 30 01:25:47.145129 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.145106 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" Oct 30 01:25:47.145358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.145102 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" event={"ID":"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d","Type":"ContainerDied","Data":"46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b"} Oct 30 01:25:47.145358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.145265 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd" event={"ID":"fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d","Type":"ContainerDied","Data":"da3f2a1815f009c2b569585ace9a8036bc549bd2aa96ae77fa9b1c807bde7adc"} Oct 30 01:25:47.145358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.145302 4905 scope.go:122] "RemoveContainer" containerID="46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b" Oct 30 01:25:47.156619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.156594 4905 scope.go:122] "RemoveContainer" containerID="9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c" Oct 30 01:25:47.164995 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.164969 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.164999 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.165014 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.165027 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.165041 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvwqp\" (UniqueName: \"kubernetes.io/projected/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-kube-api-access-nvwqp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.165056 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.165069 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.165081 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.165105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.165094 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.167602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.167585 4905 scope.go:122] "RemoveContainer" containerID="46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b" Oct 30 01:25:47.167977 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:47.167939 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b\": container with ID starting with 46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b not found: ID does not exist" containerID="46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b" Oct 30 01:25:47.168067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.167979 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b"} err="failed to get container status \"46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b\": rpc error: code = NotFound desc = could not find container \"46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b\": container with ID starting with 46876578efe1def9480d2d44c5cc1b85a7e028d5129640a3a72af2c42c4ce55b not found: ID does not exist" Oct 30 01:25:47.168067 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.168005 4905 scope.go:122] "RemoveContainer" containerID="9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c" Oct 30 01:25:47.168338 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:47.168314 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c\": container with ID starting with 9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c not found: ID does not exist" containerID="9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c" Oct 30 01:25:47.168429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.168345 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c"} err="failed to get container status \"9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c\": rpc error: code = NotFound desc = could not find container \"9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c\": container with ID starting with 9c28ac4a474d1a91c86cd665dd3e0488cd0de9f1c746b44863dc9a5a88a0611c not found: ID does not exist" Oct 30 01:25:47.206406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.206369 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd"] Oct 30 01:25:47.215674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.215642 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-11fbe8-openshift-default-fcf4879f7-b74hd"] Oct 30 01:25:47.456820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.456733 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr"] Oct 30 01:25:47.457361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.457344 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" Oct 30 01:25:47.457361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.457359 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" Oct 30 01:25:47.457561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.457404 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" Oct 30 01:25:47.457561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.457410 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" Oct 30 01:25:47.457561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.457492 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" Oct 30 01:25:47.457561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.457509 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" containerName="istio-proxy" Oct 30 01:25:47.462396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.462373 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.480708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.480682 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6c37c9-dockercfg-zbggh\"" Oct 30 01:25:47.482744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.482719 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr"] Oct 30 01:25:47.574719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.574667 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6c37c9\"" Oct 30 01:25:47.575495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.575459 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqh9\" (UniqueName: \"kubernetes.io/projected/c5d64af5-b3d4-4578-a926-2f383a11ac5f-kube-api-access-rzqh9\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.575794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.575770 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d64af5-b3d4-4578-a926-2f383a11ac5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.621298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.620904 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm"] Oct 30 01:25:47.630973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.630946 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm"] Oct 30 01:25:47.631151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.631055 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:47.676679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.676645 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d64af5-b3d4-4578-a926-2f383a11ac5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.676849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.676759 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqh9\" (UniqueName: \"kubernetes.io/projected/c5d64af5-b3d4-4578-a926-2f383a11ac5f-kube-api-access-rzqh9\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.677355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.677330 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d64af5-b3d4-4578-a926-2f383a11ac5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.698970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.698939 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqh9\" (UniqueName: \"kubernetes.io/projected/c5d64af5-b3d4-4578-a926-2f383a11ac5f-kube-api-access-rzqh9\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.776437 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.776395 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:47.777895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.777868 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvfb\" (UniqueName: \"kubernetes.io/projected/92c7c661-5317-4836-b4d3-6b7178b3dc61-kube-api-access-jcvfb\") pod \"payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:47.777975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.777919 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c7c661-5317-4836-b4d3-6b7178b3dc61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:47.879580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.879541 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvfb\" (UniqueName: \"kubernetes.io/projected/92c7c661-5317-4836-b4d3-6b7178b3dc61-kube-api-access-jcvfb\") pod \"payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:47.879817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.879620 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c7c661-5317-4836-b4d3-6b7178b3dc61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:47.880845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.880820 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c7c661-5317-4836-b4d3-6b7178b3dc61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:47.902719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.902686 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvfb\" (UniqueName: \"kubernetes.io/projected/92c7c661-5317-4836-b4d3-6b7178b3dc61-kube-api-access-jcvfb\") pod \"payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:47.949001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:47.948963 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:48.093184 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:48.093146 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr"] Oct 30 01:25:48.158716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:48.158675 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" event={"ID":"c5d64af5-b3d4-4578-a926-2f383a11ac5f","Type":"ContainerStarted","Data":"49535c98f0ce9d49e7df4b56dd896220e8c6bc3748a326918c6b66b5ebdeeccf"} Oct 30 01:25:48.291065 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:25:48.291034 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c7c661_5317_4836_b4d3_6b7178b3dc61.slice/crio-376475ee8c7d6c02ba99f1ad1220c163dd8d79cd0066f94666f575161aa9d0bd WatchSource:0}: Error finding container 376475ee8c7d6c02ba99f1ad1220c163dd8d79cd0066f94666f575161aa9d0bd: Status 404 returned error can't find the container with id 376475ee8c7d6c02ba99f1ad1220c163dd8d79cd0066f94666f575161aa9d0bd Oct 30 01:25:48.291495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:48.291475 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm"] Oct 30 01:25:49.167781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:49.167746 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" event={"ID":"c5d64af5-b3d4-4578-a926-2f383a11ac5f","Type":"ContainerStarted","Data":"277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e"} Oct 30 01:25:49.168248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:49.167875 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:25:49.169589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:49.169557 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" event={"ID":"92c7c661-5317-4836-b4d3-6b7178b3dc61","Type":"ContainerStarted","Data":"2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a"} Oct 30 01:25:49.169718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:49.169614 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" event={"ID":"92c7c661-5317-4836-b4d3-6b7178b3dc61","Type":"ContainerStarted","Data":"376475ee8c7d6c02ba99f1ad1220c163dd8d79cd0066f94666f575161aa9d0bd"} Oct 30 01:25:49.191352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:49.191320 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d" path="/var/lib/kubelet/pods/fb28b3ea-2dd1-4247-9ae2-a635bd9dd10d/volumes" Oct 30 01:25:49.209212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:49.209158 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" podStartSLOduration=2.2091412249999998 podStartE2EDuration="2.209141225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:49.208490863 +0000 UTC m=+2834.151664057" watchObservedRunningTime="2026-10-30 01:25:49.209141225 +0000 UTC m=+2834.152314419" Oct 30 01:25:49.251860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:49.251809 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" podStartSLOduration=2.251793804 podStartE2EDuration="2.251793804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:49.24841306 +0000 UTC m=+2834.191586265" watchObservedRunningTime="2026-10-30 01:25:49.251793804 +0000 UTC m=+2834.194966996" Oct 30 01:25:50.176715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:50.176683 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:25:54.347893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.347850 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554"] Oct 30 01:25:54.348389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.348104 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" podUID="dc48de16-a096-4ab3-aa7b-d15520e4ed6b" containerName="payload-processing" containerID="cri-o://7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8" gracePeriod=30 Oct 30 01:25:54.364736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.364707 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6"] Oct 30 01:25:54.364970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.364933 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" podUID="e3b9fab1-468c-451d-b380-4870b39a6bad" containerName="payload-pre-processing" containerID="cri-o://d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf" gracePeriod=30 Oct 30 01:25:54.814542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.813760 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:25:54.878742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.874263 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-kube-api-access-fnkjg\" (UniqueName: \"kubernetes.io/projected/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-kube-api-access-fnkjg\") pod \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " Oct 30 01:25:54.878742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.874611 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-plugins-config-volume\") pod \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\" (UID: \"dc48de16-a096-4ab3-aa7b-d15520e4ed6b\") " Oct 30 01:25:54.878742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.878646 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-plugins-config-volume" pod "dc48de16-a096-4ab3-aa7b-d15520e4ed6b" (UID: "dc48de16-a096-4ab3-aa7b-d15520e4ed6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:54.893006 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.892752 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-kube-api-access-fnkjg" pod "dc48de16-a096-4ab3-aa7b-d15520e4ed6b" (UID: "dc48de16-a096-4ab3-aa7b-d15520e4ed6b"). InnerVolumeSpecName "kube-api-access-fnkjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:54.912854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.912824 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:25:54.993227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.976069 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b9fab1-468c-451d-b380-4870b39a6bad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b9fab1-468c-451d-b380-4870b39a6bad-plugins-config-volume\") pod \"e3b9fab1-468c-451d-b380-4870b39a6bad\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " Oct 30 01:25:54.993227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.976307 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b9fab1-468c-451d-b380-4870b39a6bad-kube-api-access-pdp9x\" (UniqueName: \"kubernetes.io/projected/e3b9fab1-468c-451d-b380-4870b39a6bad-kube-api-access-pdp9x\") pod \"e3b9fab1-468c-451d-b380-4870b39a6bad\" (UID: \"e3b9fab1-468c-451d-b380-4870b39a6bad\") " Oct 30 01:25:54.993227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.976867 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:54.993227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.976889 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnkjg\" (UniqueName: \"kubernetes.io/projected/dc48de16-a096-4ab3-aa7b-d15520e4ed6b-kube-api-access-fnkjg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:54.993227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:54.983203 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b9fab1-468c-451d-b380-4870b39a6bad-plugins-config-volume" pod "e3b9fab1-468c-451d-b380-4870b39a6bad" (UID: "e3b9fab1-468c-451d-b380-4870b39a6bad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:55.103568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.103474 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b9fab1-468c-451d-b380-4870b39a6bad-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:55.107392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.106659 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b9fab1-468c-451d-b380-4870b39a6bad-kube-api-access-pdp9x" pod "e3b9fab1-468c-451d-b380-4870b39a6bad" (UID: "e3b9fab1-468c-451d-b380-4870b39a6bad"). InnerVolumeSpecName "kube-api-access-pdp9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:55.204305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.204268 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdp9x\" (UniqueName: \"kubernetes.io/projected/e3b9fab1-468c-451d-b380-4870b39a6bad-kube-api-access-pdp9x\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:25:55.209243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.209214 4905 generic.go:356] "Generic (PLEG): container finished" podID="e3b9fab1-468c-451d-b380-4870b39a6bad" containerID="d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf" exitCode=0 Oct 30 01:25:55.209390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.209285 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" Oct 30 01:25:55.209390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.209298 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" event={"ID":"e3b9fab1-468c-451d-b380-4870b39a6bad","Type":"ContainerDied","Data":"d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf"} Oct 30 01:25:55.209390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.209355 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6" event={"ID":"e3b9fab1-468c-451d-b380-4870b39a6bad","Type":"ContainerDied","Data":"108cb377fd60a9d7463e1a403febfeb52698e63019822a7d65d2ba5ae9f024c5"} Oct 30 01:25:55.209390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.209382 4905 scope.go:122] "RemoveContainer" containerID="d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf" Oct 30 01:25:55.213854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.213827 4905 generic.go:356] "Generic (PLEG): container finished" podID="dc48de16-a096-4ab3-aa7b-d15520e4ed6b" containerID="7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8" exitCode=0 Oct 30 01:25:55.213954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.213860 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" event={"ID":"dc48de16-a096-4ab3-aa7b-d15520e4ed6b","Type":"ContainerDied","Data":"7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8"} Oct 30 01:25:55.213954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.213885 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" Oct 30 01:25:55.213954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.213898 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554" event={"ID":"dc48de16-a096-4ab3-aa7b-d15520e4ed6b","Type":"ContainerDied","Data":"6540fe1126795eefab3646d41a5ddcbb7fb59e3ceb8c45d270c21882195daf48"} Oct 30 01:25:55.218275 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:55.218245 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc48de16_a096_4ab3_aa7b_d15520e4ed6b.slice/crio-6540fe1126795eefab3646d41a5ddcbb7fb59e3ceb8c45d270c21882195daf48\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc48de16_a096_4ab3_aa7b_d15520e4ed6b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:25:55.227080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.227064 4905 scope.go:122] "RemoveContainer" containerID="d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf" Oct 30 01:25:55.227386 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:55.227367 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf\": container with ID starting with d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf not found: ID does not exist" containerID="d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf" Oct 30 01:25:55.227448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.227395 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf"} err="failed to get container status \"d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf\": rpc error: code = NotFound desc = could not find container \"d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf\": container with ID starting with d6085d0e3bb116d2445f25f514556a10c280d98bf3b83610b5ff69ce7eaf4ccf not found: ID does not exist" Oct 30 01:25:55.227448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.227414 4905 scope.go:122] "RemoveContainer" containerID="7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8" Oct 30 01:25:55.241035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.241015 4905 scope.go:122] "RemoveContainer" containerID="7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8" Oct 30 01:25:55.241347 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:25:55.241323 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8\": container with ID starting with 7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8 not found: ID does not exist" containerID="7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8" Oct 30 01:25:55.241451 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.241353 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8"} err="failed to get container status \"7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8\": rpc error: code = NotFound desc = could not find container \"7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8\": container with ID starting with 7f2a0d4263b5ecf17e9aa1c26e9d7601f09b5d0aeb3c62d4851da13aa1fa99f8 not found: ID does not exist" Oct 30 01:25:55.266674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.266603 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6"] Oct 30 01:25:55.269427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.269404 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-b84fd64ff-ffjw6"] Oct 30 01:25:55.308674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.308647 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554"] Oct 30 01:25:55.311558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:55.311533 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-6c46dc8d4-nc554"] Oct 30 01:25:57.190789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:57.190755 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc48de16-a096-4ab3-aa7b-d15520e4ed6b" path="/var/lib/kubelet/pods/dc48de16-a096-4ab3-aa7b-d15520e4ed6b/volumes" Oct 30 01:25:57.191195 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:57.191180 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b9fab1-468c-451d-b380-4870b39a6bad" path="/var/lib/kubelet/pods/e3b9fab1-468c-451d-b380-4870b39a6bad/volumes" Oct 30 01:25:58.525966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:58.525930 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f"] Oct 30 01:25:58.526507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:25:58.526217 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" containerID="cri-o://9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae" gracePeriod=30 Oct 30 01:26:00.182075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:00.182036 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:26:01.192511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:01.192484 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:26:02.334355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.334319 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2"] Oct 30 01:26:02.335035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.335018 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e3b9fab1-468c-451d-b380-4870b39a6bad" containerName="payload-pre-processing" Oct 30 01:26:02.335096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.335036 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b9fab1-468c-451d-b380-4870b39a6bad" containerName="payload-pre-processing" Oct 30 01:26:02.335096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.335071 4905 cpu_manager.go:414] "containerMap: removing container" podUID="dc48de16-a096-4ab3-aa7b-d15520e4ed6b" containerName="payload-processing" Oct 30 01:26:02.335096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.335078 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc48de16-a096-4ab3-aa7b-d15520e4ed6b" containerName="payload-processing" Oct 30 01:26:02.335198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.335163 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b9fab1-468c-451d-b380-4870b39a6bad" containerName="payload-pre-processing" Oct 30 01:26:02.335198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.335178 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc48de16-a096-4ab3-aa7b-d15520e4ed6b" containerName="payload-processing" Oct 30 01:26:02.340384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.340355 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.376407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.376365 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2"] Oct 30 01:26:02.421387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.421356 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.421607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.421395 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfhn\" (UniqueName: \"kubernetes.io/projected/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-kube-api-access-vvfhn\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.421607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.421592 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.421723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.421649 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.447513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.447482 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6c37c9-serving-cert\"" Oct 30 01:26:02.523188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.523152 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.523382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.523215 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.523382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.523311 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.523382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.523330 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfhn\" (UniqueName: \"kubernetes.io/projected/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-kube-api-access-vvfhn\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.525623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.525597 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.527964 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.527925 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.528158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.528139 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.544386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.544356 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfhn\" (UniqueName: \"kubernetes.io/projected/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-kube-api-access-vvfhn\") pod \"maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.654885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.654807 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:02.956437 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:26:02.956395 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbacdd4_1e3d_4c95_ad51_655a5bb8a8d5.slice/crio-c8adb174bd051b7f4e6b5c7dfc659ba71f98c1b59b69ef41c838bf236c15bb7b WatchSource:0}: Error finding container c8adb174bd051b7f4e6b5c7dfc659ba71f98c1b59b69ef41c838bf236c15bb7b: Status 404 returned error can't find the container with id c8adb174bd051b7f4e6b5c7dfc659ba71f98c1b59b69ef41c838bf236c15bb7b Oct 30 01:26:02.956736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.956714 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2"] Oct 30 01:26:02.958010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:02.957990 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:26:03.280680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.280640 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" event={"ID":"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5","Type":"ContainerStarted","Data":"c8adb174bd051b7f4e6b5c7dfc659ba71f98c1b59b69ef41c838bf236c15bb7b"} Oct 30 01:26:03.418417 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.418380 4905 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:26:03.418921 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.418439 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:26:03.861074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.861047 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 01:26:03.945289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945255 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-token\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-token\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945295 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-credential-socket\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945327 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-podinfo\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945366 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-envoy\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945427 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-certs\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945465 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-kube-api-access-psl4t\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-kube-api-access-psl4t\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945587 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-socket\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945602 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-credential-socket" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:03.945824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945615 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istiod-ca-cert\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.945824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.945699 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-data\") pod \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\" (UID: \"ed9eed20-b131-46a4-84e2-e1dfa732bfdb\") " Oct 30 01:26:03.946505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.946355 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-socket" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:03.946700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.946549 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:03.946700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.946566 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istiod-ca-cert" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:03.946803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.946701 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-data" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:03.946895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.946874 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-certs" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:03.948059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.948032 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-token" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:03.948379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.948355 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-kube-api-access-psl4t" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "kube-api-access-psl4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:03.948906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.948883 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-podinfo" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:26:03.949105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:03.949089 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-envoy" pod "ed9eed20-b131-46a4-84e2-e1dfa732bfdb" (UID: "ed9eed20-b131-46a4-84e2-e1dfa732bfdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:04.048267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048228 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psl4t\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-kube-api-access-psl4t\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.048267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048262 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.048267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048272 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.048560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048281 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.048560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048289 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.048560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048297 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.048560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048304 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.048560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.048314 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9eed20-b131-46a4-84e2-e1dfa732bfdb-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:04.288879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.288841 4905 generic.go:356] "Generic (PLEG): container finished" podID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerID="9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae" exitCode=0 Oct 30 01:26:04.289082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.288895 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" Oct 30 01:26:04.289082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.288929 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" event={"ID":"ed9eed20-b131-46a4-84e2-e1dfa732bfdb","Type":"ContainerDied","Data":"9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae"} Oct 30 01:26:04.289082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.288993 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f" event={"ID":"ed9eed20-b131-46a4-84e2-e1dfa732bfdb","Type":"ContainerDied","Data":"5633d7d556f15d30803faf8b77084971cec2f6d8865a44e3587e61bb997d5976"} Oct 30 01:26:04.289082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.289017 4905 scope.go:122] "RemoveContainer" containerID="9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae" Oct 30 01:26:04.291053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.291026 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" event={"ID":"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5","Type":"ContainerStarted","Data":"a52a300f50a54425203d4110f06e2f1256e630a9f0740f3c7292a80a8a3755bd"} Oct 30 01:26:04.291173 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.291123 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:04.301148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.301131 4905 scope.go:122] "RemoveContainer" containerID="637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180" Oct 30 01:26:04.310398 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.310369 4905 scope.go:122] "RemoveContainer" containerID="9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae" Oct 30 01:26:04.310691 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:04.310673 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae\": container with ID starting with 9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae not found: ID does not exist" containerID="9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae" Oct 30 01:26:04.310747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.310700 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae"} err="failed to get container status \"9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae\": rpc error: code = NotFound desc = could not find container \"9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae\": container with ID starting with 9ed924092a8d6be4965f48d3e2c421923aed755021c8d6e9d58d31b1ee2fbfae not found: ID does not exist" Oct 30 01:26:04.310747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.310719 4905 scope.go:122] "RemoveContainer" containerID="637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180" Oct 30 01:26:04.310979 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:04.310961 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180\": container with ID starting with 637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180 not found: ID does not exist" containerID="637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180" Oct 30 01:26:04.311046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.310985 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180"} err="failed to get container status \"637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180\": rpc error: code = NotFound desc = could not find container \"637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180\": container with ID starting with 637735fab6f82e0db5806f00ff50229c38f88db5fc7a899bd5380b3c86784180 not found: ID does not exist" Oct 30 01:26:04.345768 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.345735 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f"] Oct 30 01:26:04.348680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.348651 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-25t7f"] Oct 30 01:26:04.391630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:04.391577 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" podStartSLOduration=1.9963695160000001 podStartE2EDuration="2.391561413s" totalImagesPullingTime="395.191897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:04.389072399 +0000 UTC m=+2849.332245587" watchObservedRunningTime="2026-10-30 01:26:04.391561413 +0000 UTC m=+2849.334734609" Oct 30 01:26:05.190575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:05.190539 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" path="/var/lib/kubelet/pods/ed9eed20-b131-46a4-84e2-e1dfa732bfdb/volumes" Oct 30 01:26:10.310792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:10.310766 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:26:12.473551 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:12.471918 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:26:13.063257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:13.062677 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw"] Oct 30 01:26:13.096218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:13.096184 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7"] Oct 30 01:26:18.063452 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:18.063392 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" podUID="1e90d226-6961-4cd5-8132-77c006205f67" containerName="payload-processing" containerID="cri-o://cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736" gracePeriod=25 Oct 30 01:26:18.096554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:18.096487 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" podUID="1df68de3-3599-4f4c-9e25-7e85f942467d" containerName="payload-pre-processing" containerID="cri-o://37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e" gracePeriod=25 Oct 30 01:26:19.427510 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:19.427477 4905 generic.go:356] "Generic (PLEG): container finished" podID="72332109-52dc-4c45-9507-a45b122b54d0" containerID="2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d" exitCode=0 Oct 30 01:26:19.427988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:19.427598 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" event={"ID":"72332109-52dc-4c45-9507-a45b122b54d0","Type":"ContainerDied","Data":"2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d"} Oct 30 01:26:19.428036 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:19.427987 4905 scope.go:122] "RemoveContainer" containerID="2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d" Oct 30 01:26:19.429464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:19.429433 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:26:19.429553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:19.429539 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:26:19.429601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:19.429578 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:26:20.159033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:20.158993 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" podUID="1e90d226-6961-4cd5-8132-77c006205f67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:20.159221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:20.158987 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" podUID="1df68de3-3599-4f4c-9e25-7e85f942467d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:20.457772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:20.457678 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" event={"ID":"72332109-52dc-4c45-9507-a45b122b54d0","Type":"ContainerStarted","Data":"ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b"} Oct 30 01:26:20.824128 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:20.824091 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:26:20.824316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:20.824280 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 01:26:20.824370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:20.824338 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 01:26:21.015946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:21.015904 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm"] Oct 30 01:26:21.054321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:21.054289 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr"] Oct 30 01:26:21.824159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:21.824118 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 01:26:21.824624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:21.824181 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 01:26:22.824359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:22.824326 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 01:26:22.824881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:22.824380 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 01:26:23.824326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:23.824291 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 01:26:23.824511 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:23.824352 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 01:26:24.831547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:24.828617 4905 patch_prober.go:29] interesting pod/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 01:26:24.831547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:24.828681 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 01:26:25.832342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:25.832306 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:26:25.832901 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:25.832842 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:26:25.833731 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:25.833706 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:26:26.018390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.018325 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" podUID="92c7c661-5317-4836-b4d3-6b7178b3dc61" containerName="payload-processing" containerID="cri-o://2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a" gracePeriod=25 Oct 30 01:26:26.065891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.055447 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" podUID="c5d64af5-b3d4-4578-a926-2f383a11ac5f" containerName="payload-pre-processing" containerID="cri-o://277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e" gracePeriod=25 Oct 30 01:26:26.360015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.359933 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:26:26.392274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.392242 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c7c661-5317-4836-b4d3-6b7178b3dc61-kube-api-access-jcvfb\" (UniqueName: \"kubernetes.io/projected/92c7c661-5317-4836-b4d3-6b7178b3dc61-kube-api-access-jcvfb\") pod \"92c7c661-5317-4836-b4d3-6b7178b3dc61\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " Oct 30 01:26:26.392454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.392438 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c7c661-5317-4836-b4d3-6b7178b3dc61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c7c661-5317-4836-b4d3-6b7178b3dc61-plugins-config-volume\") pod \"92c7c661-5317-4836-b4d3-6b7178b3dc61\" (UID: \"92c7c661-5317-4836-b4d3-6b7178b3dc61\") " Oct 30 01:26:26.392892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.392860 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c7c661-5317-4836-b4d3-6b7178b3dc61-plugins-config-volume" pod "92c7c661-5317-4836-b4d3-6b7178b3dc61" (UID: "92c7c661-5317-4836-b4d3-6b7178b3dc61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:26.393041 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.393019 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c7c661-5317-4836-b4d3-6b7178b3dc61-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.394870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.394845 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c7c661-5317-4836-b4d3-6b7178b3dc61-kube-api-access-jcvfb" pod "92c7c661-5317-4836-b4d3-6b7178b3dc61" (UID: "92c7c661-5317-4836-b4d3-6b7178b3dc61"). InnerVolumeSpecName "kube-api-access-jcvfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:26.453049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.453022 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:26:26.493589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.493545 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d64af5-b3d4-4578-a926-2f383a11ac5f-kube-api-access-rzqh9\" (UniqueName: \"kubernetes.io/projected/c5d64af5-b3d4-4578-a926-2f383a11ac5f-kube-api-access-rzqh9\") pod \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " Oct 30 01:26:26.493790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.493771 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d64af5-b3d4-4578-a926-2f383a11ac5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d64af5-b3d4-4578-a926-2f383a11ac5f-plugins-config-volume\") pod \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\" (UID: \"c5d64af5-b3d4-4578-a926-2f383a11ac5f\") " Oct 30 01:26:26.494431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.494396 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvfb\" (UniqueName: \"kubernetes.io/projected/92c7c661-5317-4836-b4d3-6b7178b3dc61-kube-api-access-jcvfb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.494792 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.494770 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d64af5-b3d4-4578-a926-2f383a11ac5f-plugins-config-volume" pod "c5d64af5-b3d4-4578-a926-2f383a11ac5f" (UID: "c5d64af5-b3d4-4578-a926-2f383a11ac5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:26.496321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.496273 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d64af5-b3d4-4578-a926-2f383a11ac5f-kube-api-access-rzqh9" pod "c5d64af5-b3d4-4578-a926-2f383a11ac5f" (UID: "c5d64af5-b3d4-4578-a926-2f383a11ac5f"). InnerVolumeSpecName "kube-api-access-rzqh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:26.510895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.510871 4905 generic.go:356] "Generic (PLEG): container finished" podID="c5d64af5-b3d4-4578-a926-2f383a11ac5f" containerID="277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e" exitCode=0 Oct 30 01:26:26.511033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.510918 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" Oct 30 01:26:26.511033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.510958 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" event={"ID":"c5d64af5-b3d4-4578-a926-2f383a11ac5f","Type":"ContainerDied","Data":"277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e"} Oct 30 01:26:26.511151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.511037 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr" event={"ID":"c5d64af5-b3d4-4578-a926-2f383a11ac5f","Type":"ContainerDied","Data":"49535c98f0ce9d49e7df4b56dd896220e8c6bc3748a326918c6b66b5ebdeeccf"} Oct 30 01:26:26.511151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.511064 4905 scope.go:122] "RemoveContainer" containerID="277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e" Oct 30 01:26:26.513135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.513109 4905 generic.go:356] "Generic (PLEG): container finished" podID="92c7c661-5317-4836-b4d3-6b7178b3dc61" containerID="2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a" exitCode=0 Oct 30 01:26:26.513244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.513155 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" Oct 30 01:26:26.513244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.513172 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" event={"ID":"92c7c661-5317-4836-b4d3-6b7178b3dc61","Type":"ContainerDied","Data":"2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a"} Oct 30 01:26:26.513244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.513238 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm" event={"ID":"92c7c661-5317-4836-b4d3-6b7178b3dc61","Type":"ContainerDied","Data":"376475ee8c7d6c02ba99f1ad1220c163dd8d79cd0066f94666f575161aa9d0bd"} Oct 30 01:26:26.524496 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.524472 4905 scope.go:122] "RemoveContainer" containerID="277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e" Oct 30 01:26:26.524883 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:26.524851 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e\": container with ID starting with 277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e not found: ID does not exist" containerID="277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e" Oct 30 01:26:26.524956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.524889 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e"} err="failed to get container status \"277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e\": rpc error: code = NotFound desc = could not find container \"277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e\": container with ID starting with 277d4d5b927711e36588388da9b15d7350de516874d6a9fb43e51380e646e05e not found: ID does not exist" Oct 30 01:26:26.524956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.524912 4905 scope.go:122] "RemoveContainer" containerID="2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a" Oct 30 01:26:26.537485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.537407 4905 scope.go:122] "RemoveContainer" containerID="2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a" Oct 30 01:26:26.537827 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:26.537801 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a\": container with ID starting with 2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a not found: ID does not exist" containerID="2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a" Oct 30 01:26:26.537891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.537837 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a"} err="failed to get container status \"2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a\": rpc error: code = NotFound desc = could not find container \"2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a\": container with ID starting with 2d0074f37879ed07a3af0d9b8f981e69443b4d96ee6afbcd2c07a11054688a7a not found: ID does not exist" Oct 30 01:26:26.566430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.566408 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr"] Oct 30 01:26:26.569089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.569071 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-7b7fd5667f-shfcr"] Oct 30 01:26:26.595132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.595108 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d64af5-b3d4-4578-a926-2f383a11ac5f-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.595132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.595132 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzqh9\" (UniqueName: \"kubernetes.io/projected/c5d64af5-b3d4-4578-a926-2f383a11ac5f-kube-api-access-rzqh9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.622787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.622725 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm"] Oct 30 01:26:26.625706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:26.625688 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-5fb97b477b-dfdkm"] Oct 30 01:26:27.193652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:27.193618 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c7c661-5317-4836-b4d3-6b7178b3dc61" path="/var/lib/kubelet/pods/92c7c661-5317-4836-b4d3-6b7178b3dc61/volumes" Oct 30 01:26:27.194723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:27.194697 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d64af5-b3d4-4578-a926-2f383a11ac5f" path="/var/lib/kubelet/pods/c5d64af5-b3d4-4578-a926-2f383a11ac5f/volumes" Oct 30 01:26:30.158319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:30.158280 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" podUID="1e90d226-6961-4cd5-8132-77c006205f67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:30.158867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:30.158769 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" podUID="1df68de3-3599-4f4c-9e25-7e85f942467d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:33.061364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.061326 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b"] Oct 30 01:26:33.062227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062207 4905 cpu_manager.go:414] "containerMap: removing container" podUID="c5d64af5-b3d4-4578-a926-2f383a11ac5f" containerName="payload-pre-processing" Oct 30 01:26:33.062282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062230 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d64af5-b3d4-4578-a926-2f383a11ac5f" containerName="payload-pre-processing" Oct 30 01:26:33.062282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062247 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" Oct 30 01:26:33.062282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062252 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" Oct 30 01:26:33.062282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062262 4905 cpu_manager.go:414] "containerMap: removing container" podUID="92c7c661-5317-4836-b4d3-6b7178b3dc61" containerName="payload-processing" Oct 30 01:26:33.062282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062267 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c7c661-5317-4836-b4d3-6b7178b3dc61" containerName="payload-processing" Oct 30 01:26:33.062460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062387 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" Oct 30 01:26:33.062460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062407 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d64af5-b3d4-4578-a926-2f383a11ac5f" containerName="payload-pre-processing" Oct 30 01:26:33.062460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062418 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" Oct 30 01:26:33.062460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.062425 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c7c661-5317-4836-b4d3-6b7178b3dc61" containerName="payload-processing" Oct 30 01:26:33.064981 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.064951 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.070435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.070402 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8g7\" (UniqueName: \"kubernetes.io/projected/52ec8470-e62a-4d21-b66f-debbcfbca51f-kube-api-access-rz8g7\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.070693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.070670 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec8470-e62a-4d21-b66f-debbcfbca51f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.164018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.143987 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6c37c9-dockercfg-g4ttd\"" Oct 30 01:26:33.164018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.144096 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6c37c9\"" Oct 30 01:26:33.171215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.171088 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b"] Oct 30 01:26:33.172231 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.172197 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec8470-e62a-4d21-b66f-debbcfbca51f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.172733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.172711 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8g7\" (UniqueName: \"kubernetes.io/projected/52ec8470-e62a-4d21-b66f-debbcfbca51f-kube-api-access-rz8g7\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.173891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.173861 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec8470-e62a-4d21-b66f-debbcfbca51f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.203996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.203962 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8g7\" (UniqueName: \"kubernetes.io/projected/52ec8470-e62a-4d21-b66f-debbcfbca51f-kube-api-access-rz8g7\") pod \"payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.306659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.306627 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr"] Oct 30 01:26:33.306871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.306849 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" containerID="cri-o://3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65" gracePeriod=30 Oct 30 01:26:33.332245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.332176 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww"] Oct 30 01:26:33.333089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.333062 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" Oct 30 01:26:33.333089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.333083 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9eed20-b131-46a4-84e2-e1dfa732bfdb" containerName="istio-proxy" Oct 30 01:26:33.336136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.336117 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.357355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.357328 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww"] Oct 30 01:26:33.377653 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.377615 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cq6c\" (UniqueName: \"kubernetes.io/projected/ec3c12cb-c978-4680-8e6e-c230369fbf13-kube-api-access-9cq6c\") pod \"payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.377868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.377848 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3c12cb-c978-4680-8e6e-c230369fbf13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.478338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.478069 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:33.483791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.483762 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cq6c\" (UniqueName: \"kubernetes.io/projected/ec3c12cb-c978-4680-8e6e-c230369fbf13-kube-api-access-9cq6c\") pod \"payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.483976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.483948 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3c12cb-c978-4680-8e6e-c230369fbf13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.485014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.484992 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3c12cb-c978-4680-8e6e-c230369fbf13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.505046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.505018 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cq6c\" (UniqueName: \"kubernetes.io/projected/ec3c12cb-c978-4680-8e6e-c230369fbf13-kube-api-access-9cq6c\") pod \"payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.652996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.652894 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:33.788761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.788685 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b"] Oct 30 01:26:33.988973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:33.988944 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww"] Oct 30 01:26:34.578938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.578899 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" event={"ID":"ec3c12cb-c978-4680-8e6e-c230369fbf13","Type":"ContainerStarted","Data":"4773058a0338d81337e2d229c6a917a1084d84578938eb576574cd5aeda7c528"} Oct 30 01:26:34.579540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.578969 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:34.579540 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.578999 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" event={"ID":"ec3c12cb-c978-4680-8e6e-c230369fbf13","Type":"ContainerStarted","Data":"eab1c3b05b809150e81ebd9b4923555cd33926f32edbe460212db6009781cf6e"} Oct 30 01:26:34.580978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.580946 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" event={"ID":"52ec8470-e62a-4d21-b66f-debbcfbca51f","Type":"ContainerStarted","Data":"5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864"} Oct 30 01:26:34.581115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.581005 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:34.581115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.581028 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" event={"ID":"52ec8470-e62a-4d21-b66f-debbcfbca51f","Type":"ContainerStarted","Data":"7bde3e60f789a73111c46ce552994657ee50c9d6facb7b1fc8e7473a3c985710"} Oct 30 01:26:34.621626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.621567 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" podStartSLOduration=1.6215512159999999 podStartE2EDuration="1.621551216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:34.619359749 +0000 UTC m=+2879.562532943" watchObservedRunningTime="2026-10-30 01:26:34.621551216 +0000 UTC m=+2879.564724408" Oct 30 01:26:34.661683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:34.661623 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" podStartSLOduration=1.6616041369999999 podStartE2EDuration="1.661604137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:34.659222921 +0000 UTC m=+2879.602396116" watchObservedRunningTime="2026-10-30 01:26:34.661604137 +0000 UTC m=+2879.604777332" Oct 30 01:26:35.291167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:35.291125 4905 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:26:35.291381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:35.291194 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:26:38.435743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.435720 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:26:38.443356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.443329 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:26:38.552160 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.552125 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df68de3-3599-4f4c-9e25-7e85f942467d-kube-api-access-8xdcb\" (UniqueName: \"kubernetes.io/projected/1df68de3-3599-4f4c-9e25-7e85f942467d-kube-api-access-8xdcb\") pod \"1df68de3-3599-4f4c-9e25-7e85f942467d\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " Oct 30 01:26:38.552357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.552176 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df68de3-3599-4f4c-9e25-7e85f942467d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df68de3-3599-4f4c-9e25-7e85f942467d-plugins-config-volume\") pod \"1df68de3-3599-4f4c-9e25-7e85f942467d\" (UID: \"1df68de3-3599-4f4c-9e25-7e85f942467d\") " Oct 30 01:26:38.552357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.552201 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e90d226-6961-4cd5-8132-77c006205f67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e90d226-6961-4cd5-8132-77c006205f67-plugins-config-volume\") pod \"1e90d226-6961-4cd5-8132-77c006205f67\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " Oct 30 01:26:38.552357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.552271 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e90d226-6961-4cd5-8132-77c006205f67-kube-api-access-gq7hc\" (UniqueName: \"kubernetes.io/projected/1e90d226-6961-4cd5-8132-77c006205f67-kube-api-access-gq7hc\") pod \"1e90d226-6961-4cd5-8132-77c006205f67\" (UID: \"1e90d226-6961-4cd5-8132-77c006205f67\") " Oct 30 01:26:38.553232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.553202 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df68de3-3599-4f4c-9e25-7e85f942467d-plugins-config-volume" pod "1df68de3-3599-4f4c-9e25-7e85f942467d" (UID: "1df68de3-3599-4f4c-9e25-7e85f942467d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:38.553793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.553764 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e90d226-6961-4cd5-8132-77c006205f67-plugins-config-volume" pod "1e90d226-6961-4cd5-8132-77c006205f67" (UID: "1e90d226-6961-4cd5-8132-77c006205f67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:38.555922 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.555898 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e90d226-6961-4cd5-8132-77c006205f67-kube-api-access-gq7hc" pod "1e90d226-6961-4cd5-8132-77c006205f67" (UID: "1e90d226-6961-4cd5-8132-77c006205f67"). InnerVolumeSpecName "kube-api-access-gq7hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:38.556773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.556747 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df68de3-3599-4f4c-9e25-7e85f942467d-kube-api-access-8xdcb" pod "1df68de3-3599-4f4c-9e25-7e85f942467d" (UID: "1df68de3-3599-4f4c-9e25-7e85f942467d"). InnerVolumeSpecName "kube-api-access-8xdcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:38.591390 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.591367 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:26:38.607669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.607644 4905 generic.go:356] "Generic (PLEG): container finished" podID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerID="3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65" exitCode=0 Oct 30 01:26:38.607797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.607679 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" Oct 30 01:26:38.607797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.607724 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" event={"ID":"2b712d86-c47a-41a9-9a25-1ae45c0ba963","Type":"ContainerDied","Data":"3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65"} Oct 30 01:26:38.607797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.607787 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr" event={"ID":"2b712d86-c47a-41a9-9a25-1ae45c0ba963","Type":"ContainerDied","Data":"d38a5ef0ab942cf49fed41210089e368d370a4ce6563c1b6afd5f820986cce5c"} Oct 30 01:26:38.607938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.607805 4905 scope.go:122] "RemoveContainer" containerID="3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65" Oct 30 01:26:38.609138 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.609114 4905 generic.go:356] "Generic (PLEG): container finished" podID="1df68de3-3599-4f4c-9e25-7e85f942467d" containerID="37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e" exitCode=0 Oct 30 01:26:38.609264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.609144 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" Oct 30 01:26:38.609264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.609192 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" event={"ID":"1df68de3-3599-4f4c-9e25-7e85f942467d","Type":"ContainerDied","Data":"37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e"} Oct 30 01:26:38.609264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.609237 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7" event={"ID":"1df68de3-3599-4f4c-9e25-7e85f942467d","Type":"ContainerDied","Data":"700d5d8fe247ef1c639ad318855ce867d401a9f70176e2d869f56f3d561c8fca"} Oct 30 01:26:38.611338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.611319 4905 generic.go:356] "Generic (PLEG): container finished" podID="1e90d226-6961-4cd5-8132-77c006205f67" containerID="cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736" exitCode=0 Oct 30 01:26:38.611435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.611364 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" event={"ID":"1e90d226-6961-4cd5-8132-77c006205f67","Type":"ContainerDied","Data":"cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736"} Oct 30 01:26:38.611435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.611401 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" event={"ID":"1e90d226-6961-4cd5-8132-77c006205f67","Type":"ContainerDied","Data":"a6a0797d17cb9272d518cdebb5db046d6cc03239bf8ab5d7d7148afab1222f3d"} Oct 30 01:26:38.611551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.611454 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw" Oct 30 01:26:38.618797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.618777 4905 scope.go:122] "RemoveContainer" containerID="3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65" Oct 30 01:26:38.619248 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:38.619223 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65\": container with ID starting with 3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65 not found: ID does not exist" containerID="3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65" Oct 30 01:26:38.619339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.619257 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65"} err="failed to get container status \"3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65\": rpc error: code = NotFound desc = could not find container \"3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65\": container with ID starting with 3ae7185bbdee90152335ff72c1643138042dea995a87afeeb52ff7148f806e65 not found: ID does not exist" Oct 30 01:26:38.619339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.619274 4905 scope.go:122] "RemoveContainer" containerID="37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.644984 4905 scope.go:122] "RemoveContainer" containerID="37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:38.646263 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e\": container with ID starting with 37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e not found: ID does not exist" containerID="37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.646300 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e"} err="failed to get container status \"37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e\": rpc error: code = NotFound desc = could not find container \"37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e\": container with ID starting with 37e050c12310ba21850fc7e3277e1632b9e1d46158d962a74fc7970da2928e8e not found: ID does not exist" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.646351 4905 scope.go:122] "RemoveContainer" containerID="cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.654955 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-token\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-token\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.654996 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-podinfo\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.655023 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-credential-socket\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.655167 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-envoy\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.655244 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-data\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.655299 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-kube-api-access-hrsv8\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-kube-api-access-hrsv8\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.655365 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-certs\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.655407 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-socket\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.655478 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istiod-ca-cert\") pod \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\" (UID: \"2b712d86-c47a-41a9-9a25-1ae45c0ba963\") " Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.656029 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xdcb\" (UniqueName: \"kubernetes.io/projected/1df68de3-3599-4f4c-9e25-7e85f942467d-kube-api-access-8xdcb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.656052 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df68de3-3599-4f4c-9e25-7e85f942467d-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.656068 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e90d226-6961-4cd5-8132-77c006205f67-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.656307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.656083 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7hc\" (UniqueName: \"kubernetes.io/projected/1e90d226-6961-4cd5-8132-77c006205f67-kube-api-access-gq7hc\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.657705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.657662 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istiod-ca-cert" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:38.664445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.663776 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-credential-socket" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.664445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.664144 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-certs" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.665047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.665011 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-socket" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.665763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.665740 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-data" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.668661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.668631 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-envoy" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.669181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.669145 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-kube-api-access-hrsv8" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "kube-api-access-hrsv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:38.670642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.670608 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-token" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:38.671816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.671791 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-podinfo" pod "2b712d86-c47a-41a9-9a25-1ae45c0ba963" (UID: "2b712d86-c47a-41a9-9a25-1ae45c0ba963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:26:38.680595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.680572 4905 scope.go:122] "RemoveContainer" containerID="cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736" Oct 30 01:26:38.681011 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:38.680988 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736\": container with ID starting with cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736 not found: ID does not exist" containerID="cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736" Oct 30 01:26:38.681130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.681016 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736"} err="failed to get container status \"cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736\": rpc error: code = NotFound desc = could not find container \"cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736\": container with ID starting with cc6d76d4d47cea8ecc1fca9e715db7f7bee06ff7c170b12befa3110742606736 not found: ID does not exist" Oct 30 01:26:38.685943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.685919 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7"] Oct 30 01:26:38.688616 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.688585 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cb31e3-8bbfdc77f-tt7r7"] Oct 30 01:26:38.724021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.723995 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw"] Oct 30 01:26:38.726640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.726617 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cb31e3-5f4d79b596-76lhw"] Oct 30 01:26:38.757364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757331 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757359 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757368 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrsv8\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-kube-api-access-hrsv8\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757377 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757388 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757397 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757405 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757413 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b712d86-c47a-41a9-9a25-1ae45c0ba963-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.757639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.757421 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b712d86-c47a-41a9-9a25-1ae45c0ba963-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.961412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.961374 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr"] Oct 30 01:26:38.964971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:38.964947 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tvtvr"] Oct 30 01:26:39.192688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:39.192596 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df68de3-3599-4f4c-9e25-7e85f942467d" path="/var/lib/kubelet/pods/1df68de3-3599-4f4c-9e25-7e85f942467d/volumes" Oct 30 01:26:39.193114 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:39.193093 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e90d226-6961-4cd5-8132-77c006205f67" path="/var/lib/kubelet/pods/1e90d226-6961-4cd5-8132-77c006205f67/volumes" Oct 30 01:26:39.193603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:39.193585 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" path="/var/lib/kubelet/pods/2b712d86-c47a-41a9-9a25-1ae45c0ba963/volumes" Oct 30 01:26:42.414593 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:42.414555 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:26:44.251613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:44.251568 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9"] Oct 30 01:26:44.284339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:44.284307 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl"] Oct 30 01:26:45.589542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:45.589496 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:26:45.590069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:45.590020 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:26:49.252940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:49.252886 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" podUID="09b802cb-c008-4902-b352-fc95ca96d5a5" containerName="payload-processing" containerID="cri-o://9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f" gracePeriod=25 Oct 30 01:26:49.285546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:49.285478 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" podUID="46e8119b-10cc-4552-ae70-5b621711f586" containerName="payload-pre-processing" containerID="cri-o://018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225" gracePeriod=25 Oct 30 01:26:49.914178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:49.914141 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv"] Oct 30 01:26:49.914477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:49.914418 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" podUID="ada8efa9-0769-4886-9d93-14b541f44f69" containerName="maas-api" containerID="cri-o://7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060" gracePeriod=30 Oct 30 01:26:50.219622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.219599 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:26:50.295112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.295070 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada8efa9-0769-4886-9d93-14b541f44f69-kube-api-access-6rkpt\" (UniqueName: \"kubernetes.io/projected/ada8efa9-0769-4886-9d93-14b541f44f69-kube-api-access-6rkpt\") pod \"ada8efa9-0769-4886-9d93-14b541f44f69\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " Oct 30 01:26:50.295723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.295695 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada8efa9-0769-4886-9d93-14b541f44f69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada8efa9-0769-4886-9d93-14b541f44f69-openshift-service-ca\") pod \"ada8efa9-0769-4886-9d93-14b541f44f69\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " Oct 30 01:26:50.295893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.295876 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-metrics-certs\") pod \"ada8efa9-0769-4886-9d93-14b541f44f69\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " Oct 30 01:26:50.296107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.296092 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-maas-api-tls\") pod \"ada8efa9-0769-4886-9d93-14b541f44f69\" (UID: \"ada8efa9-0769-4886-9d93-14b541f44f69\") " Oct 30 01:26:50.302976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.302945 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada8efa9-0769-4886-9d93-14b541f44f69-openshift-service-ca" pod "ada8efa9-0769-4886-9d93-14b541f44f69" (UID: "ada8efa9-0769-4886-9d93-14b541f44f69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:50.339495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.339445 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada8efa9-0769-4886-9d93-14b541f44f69-kube-api-access-6rkpt" pod "ada8efa9-0769-4886-9d93-14b541f44f69" (UID: "ada8efa9-0769-4886-9d93-14b541f44f69"). InnerVolumeSpecName "kube-api-access-6rkpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:50.340795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.340771 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-maas-api-tls" pod "ada8efa9-0769-4886-9d93-14b541f44f69" (UID: "ada8efa9-0769-4886-9d93-14b541f44f69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:50.342070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.342041 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-metrics-certs" pod "ada8efa9-0769-4886-9d93-14b541f44f69" (UID: "ada8efa9-0769-4886-9d93-14b541f44f69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:50.398265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.398233 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:50.398265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.398263 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkpt\" (UniqueName: \"kubernetes.io/projected/ada8efa9-0769-4886-9d93-14b541f44f69-kube-api-access-6rkpt\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:50.398265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.398272 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada8efa9-0769-4886-9d93-14b541f44f69-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:50.398543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.398281 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada8efa9-0769-4886-9d93-14b541f44f69-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:26:50.425715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.425689 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:26:50.677282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.677250 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:26:50.693943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.693916 4905 generic.go:356] "Generic (PLEG): container finished" podID="ada8efa9-0769-4886-9d93-14b541f44f69" containerID="7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060" exitCode=0 Oct 30 01:26:50.694096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.693960 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" Oct 30 01:26:50.694096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.694002 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" event={"ID":"ada8efa9-0769-4886-9d93-14b541f44f69","Type":"ContainerDied","Data":"7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060"} Oct 30 01:26:50.694096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.694065 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv" event={"ID":"ada8efa9-0769-4886-9d93-14b541f44f69","Type":"ContainerDied","Data":"a8a245f9e7620f7aee4e2e357421432a15c52655b438c30a498a474b517c8de3"} Oct 30 01:26:50.694278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.694095 4905 scope.go:122] "RemoveContainer" containerID="7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060" Oct 30 01:26:50.704805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.704790 4905 scope.go:122] "RemoveContainer" containerID="7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060" Oct 30 01:26:50.705041 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:26:50.705019 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060\": container with ID starting with 7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060 not found: ID does not exist" containerID="7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060" Oct 30 01:26:50.705120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.705054 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060"} err="failed to get container status \"7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060\": rpc error: code = NotFound desc = could not find container \"7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060\": container with ID starting with 7156f354e008306a32f731fcac8e043776342dcea021ade5e67e0353571a7060 not found: ID does not exist" Oct 30 01:26:50.766871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.766831 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv"] Oct 30 01:26:50.774053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.774027 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ad00a0-567f5fb777-bk8mv"] Oct 30 01:26:50.880045 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:50.880013 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww_ec3c12cb-c978-4680-8e6e-c230369fbf13/payload-processing/0.log" Oct 30 01:26:51.191596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:51.191482 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada8efa9-0769-4886-9d93-14b541f44f69" path="/var/lib/kubelet/pods/ada8efa9-0769-4886-9d93-14b541f44f69/volumes" Oct 30 01:26:57.561638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:57.561599 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm"] Oct 30 01:26:57.562145 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:57.561864 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" containerID="cri-o://606cc8fdd96b924ebda96667ec9e976411a15db2e74a5760892b5e778d8db0c7" gracePeriod=30 Oct 30 01:26:57.838982 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:57.838903 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" podUID="09b802cb-c008-4902-b352-fc95ca96d5a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:57.839168 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:26:57.838906 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" podUID="46e8119b-10cc-4552-ae70-5b621711f586" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:02.787053 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.787007 4905 generic.go:356] "Generic (PLEG): container finished" podID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerID="606cc8fdd96b924ebda96667ec9e976411a15db2e74a5760892b5e778d8db0c7" exitCode=0 Oct 30 01:27:02.787600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.787068 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" event={"ID":"e5e381e2-5b6b-49af-8fe2-5ba37a189955","Type":"ContainerDied","Data":"606cc8fdd96b924ebda96667ec9e976411a15db2e74a5760892b5e778d8db0c7"} Oct 30 01:27:02.787600 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.787212 4905 scope.go:122] "RemoveContainer" containerID="8251a58ab9944e283df7ed3eb80d2d678d314344ce1baa56d14ce1953d3de845" Oct 30 01:27:02.840967 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.840896 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:27:02.955575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955544 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-credential-socket\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.955790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955600 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-socket\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.955790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955640 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-token\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-token\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.955790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955667 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-data\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.955790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955734 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-podinfo\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.956021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955832 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-credential-socket" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:02.956021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955848 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-envoy\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.956021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955914 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-socket" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:02.956021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.955986 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istiod-ca-cert\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.956180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.956043 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-certs\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.956180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.956092 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-kube-api-access-wh24z\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-kube-api-access-wh24z\") pod \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\" (UID: \"e5e381e2-5b6b-49af-8fe2-5ba37a189955\") " Oct 30 01:27:02.956273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.956248 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-certs" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:02.956378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.956360 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istiod-ca-cert" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:02.956718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.956691 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-data" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:02.957142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.957013 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:02.957142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.957036 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:02.957142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.957049 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:02.957142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.957064 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:02.958666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.958626 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-token" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:02.958886 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.958863 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-podinfo" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:27:02.959066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.959046 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-kube-api-access-wh24z" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "kube-api-access-wh24z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:02.959416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:02.959395 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-envoy" pod "e5e381e2-5b6b-49af-8fe2-5ba37a189955" (UID: "e5e381e2-5b6b-49af-8fe2-5ba37a189955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:03.039237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.039197 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz"] Oct 30 01:27:03.058049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.058013 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:03.058049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.058048 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:03.058285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.058062 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:03.058285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.058074 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e381e2-5b6b-49af-8fe2-5ba37a189955-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:03.058285 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.058089 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh24z\" (UniqueName: \"kubernetes.io/projected/e5e381e2-5b6b-49af-8fe2-5ba37a189955-kube-api-access-wh24z\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:03.796181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.796150 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" Oct 30 01:27:03.796181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.796176 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm" event={"ID":"e5e381e2-5b6b-49af-8fe2-5ba37a189955","Type":"ContainerDied","Data":"9d99a8de0fe67fa398c23530469b52b5ec35556e040678000c36e8bcfc449c32"} Oct 30 01:27:03.796794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.796232 4905 scope.go:122] "RemoveContainer" containerID="606cc8fdd96b924ebda96667ec9e976411a15db2e74a5760892b5e778d8db0c7" Oct 30 01:27:03.865159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.865121 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm"] Oct 30 01:27:03.878787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:03.878752 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-fn9gm"] Oct 30 01:27:05.197146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:05.197076 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" path="/var/lib/kubelet/pods/e5e381e2-5b6b-49af-8fe2-5ba37a189955/volumes" Oct 30 01:27:05.983120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:05.983086 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g"] Oct 30 01:27:05.983338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:05.983302 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" podUID="7e1f673b-e586-44f2-acc4-d67c2fe94bef" containerName="payload-processing" containerID="cri-o://76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e" gracePeriod=30 Oct 30 01:27:06.000159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.000122 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9"] Oct 30 01:27:06.000401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.000371 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" podUID="736ddaa0-2e2d-42c8-8277-0f9ffe3da974" containerName="payload-pre-processing" containerID="cri-o://0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d" gracePeriod=30 Oct 30 01:27:06.298030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.297997 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:27:06.394049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.392294 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1f673b-e586-44f2-acc4-d67c2fe94bef-kube-api-access-79trm\" (UniqueName: \"kubernetes.io/projected/7e1f673b-e586-44f2-acc4-d67c2fe94bef-kube-api-access-79trm\") pod \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " Oct 30 01:27:06.394049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.392383 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1f673b-e586-44f2-acc4-d67c2fe94bef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1f673b-e586-44f2-acc4-d67c2fe94bef-plugins-config-volume\") pod \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\" (UID: \"7e1f673b-e586-44f2-acc4-d67c2fe94bef\") " Oct 30 01:27:06.394049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.393200 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1f673b-e586-44f2-acc4-d67c2fe94bef-plugins-config-volume" pod "7e1f673b-e586-44f2-acc4-d67c2fe94bef" (UID: "7e1f673b-e586-44f2-acc4-d67c2fe94bef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:06.396101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.395693 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:27:06.402086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.402054 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1f673b-e586-44f2-acc4-d67c2fe94bef-kube-api-access-79trm" pod "7e1f673b-e586-44f2-acc4-d67c2fe94bef" (UID: "7e1f673b-e586-44f2-acc4-d67c2fe94bef"). InnerVolumeSpecName "kube-api-access-79trm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:06.493563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.493513 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-plugins-config-volume\") pod \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " Oct 30 01:27:06.493746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.493603 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-kube-api-access-9tckx\" (UniqueName: \"kubernetes.io/projected/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-kube-api-access-9tckx\") pod \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\" (UID: \"736ddaa0-2e2d-42c8-8277-0f9ffe3da974\") " Oct 30 01:27:06.493855 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.493833 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-plugins-config-volume" pod "736ddaa0-2e2d-42c8-8277-0f9ffe3da974" (UID: "736ddaa0-2e2d-42c8-8277-0f9ffe3da974"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:06.493983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.493969 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79trm\" (UniqueName: \"kubernetes.io/projected/7e1f673b-e586-44f2-acc4-d67c2fe94bef-kube-api-access-79trm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:06.494026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.493986 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1f673b-e586-44f2-acc4-d67c2fe94bef-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:06.494026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.493995 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:06.495828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.495781 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-kube-api-access-9tckx" pod "736ddaa0-2e2d-42c8-8277-0f9ffe3da974" (UID: "736ddaa0-2e2d-42c8-8277-0f9ffe3da974"). InnerVolumeSpecName "kube-api-access-9tckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:06.595930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.595883 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tckx\" (UniqueName: \"kubernetes.io/projected/736ddaa0-2e2d-42c8-8277-0f9ffe3da974-kube-api-access-9tckx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:06.815842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.815751 4905 generic.go:356] "Generic (PLEG): container finished" podID="7e1f673b-e586-44f2-acc4-d67c2fe94bef" containerID="76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e" exitCode=0 Oct 30 01:27:06.815842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.815806 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" Oct 30 01:27:06.816086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.815837 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" event={"ID":"7e1f673b-e586-44f2-acc4-d67c2fe94bef","Type":"ContainerDied","Data":"76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e"} Oct 30 01:27:06.816086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.815902 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g" event={"ID":"7e1f673b-e586-44f2-acc4-d67c2fe94bef","Type":"ContainerDied","Data":"f5286467daa176883768a01f8378a1b5d431c959820e2615e3e174e106237657"} Oct 30 01:27:06.816086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.815934 4905 scope.go:122] "RemoveContainer" containerID="76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e" Oct 30 01:27:06.817228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.817211 4905 generic.go:356] "Generic (PLEG): container finished" podID="736ddaa0-2e2d-42c8-8277-0f9ffe3da974" containerID="0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d" exitCode=0 Oct 30 01:27:06.817333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.817253 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" Oct 30 01:27:06.817333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.817302 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" event={"ID":"736ddaa0-2e2d-42c8-8277-0f9ffe3da974","Type":"ContainerDied","Data":"0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d"} Oct 30 01:27:06.817436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.817352 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9" event={"ID":"736ddaa0-2e2d-42c8-8277-0f9ffe3da974","Type":"ContainerDied","Data":"b1b5c04b99c83f582dc4cb190c71fee55962cd4f7e1f07f61144825340878e67"} Oct 30 01:27:06.830788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.828832 4905 scope.go:122] "RemoveContainer" containerID="76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e" Oct 30 01:27:06.831817 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:06.831789 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e\": container with ID starting with 76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e not found: ID does not exist" containerID="76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e" Oct 30 01:27:06.831935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.831826 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e"} err="failed to get container status \"76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e\": rpc error: code = NotFound desc = could not find container \"76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e\": container with ID starting with 76152ee42ae1a9558c6e916f94ebcf802a2b9a11a0bdbadd38dcc0aa5183e76e not found: ID does not exist" Oct 30 01:27:06.831935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.831847 4905 scope.go:122] "RemoveContainer" containerID="0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d" Oct 30 01:27:06.841239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.841217 4905 scope.go:122] "RemoveContainer" containerID="0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d" Oct 30 01:27:06.841488 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:06.841470 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d\": container with ID starting with 0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d not found: ID does not exist" containerID="0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d" Oct 30 01:27:06.841564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.841495 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d"} err="failed to get container status \"0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d\": rpc error: code = NotFound desc = could not find container \"0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d\": container with ID starting with 0886a6f42dffdd804b2a0ce2b1565fca4744048c2dc3de64fa109620d44ccb8d not found: ID does not exist" Oct 30 01:27:06.878623 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.878588 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g"] Oct 30 01:27:06.908714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.907022 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0519f6-6975498f48-vd42g"] Oct 30 01:27:06.936178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.936144 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9"] Oct 30 01:27:06.939631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:06.939606 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0519f6-799cc955f-lkhd9"] Oct 30 01:27:07.190876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:07.190784 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736ddaa0-2e2d-42c8-8277-0f9ffe3da974" path="/var/lib/kubelet/pods/736ddaa0-2e2d-42c8-8277-0f9ffe3da974/volumes" Oct 30 01:27:07.191303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:07.191281 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1f673b-e586-44f2-acc4-d67c2fe94bef" path="/var/lib/kubelet/pods/7e1f673b-e586-44f2-acc4-d67c2fe94bef/volumes" Oct 30 01:27:07.839299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:07.839258 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" podUID="09b802cb-c008-4902-b352-fc95ca96d5a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:07.839715 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:07.839317 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" podUID="46e8119b-10cc-4552-ae70-5b621711f586" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:09.582361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.582336 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:27:09.611080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.611058 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:27:09.639256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.639230 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e8119b-10cc-4552-ae70-5b621711f586-kube-api-access-w6wfb\" (UniqueName: \"kubernetes.io/projected/46e8119b-10cc-4552-ae70-5b621711f586-kube-api-access-w6wfb\") pod \"46e8119b-10cc-4552-ae70-5b621711f586\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " Oct 30 01:27:09.639418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.639270 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b802cb-c008-4902-b352-fc95ca96d5a5-kube-api-access-f64wz\" (UniqueName: \"kubernetes.io/projected/09b802cb-c008-4902-b352-fc95ca96d5a5-kube-api-access-f64wz\") pod \"09b802cb-c008-4902-b352-fc95ca96d5a5\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " Oct 30 01:27:09.639418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.639328 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e8119b-10cc-4552-ae70-5b621711f586-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e8119b-10cc-4552-ae70-5b621711f586-plugins-config-volume\") pod \"46e8119b-10cc-4552-ae70-5b621711f586\" (UID: \"46e8119b-10cc-4552-ae70-5b621711f586\") " Oct 30 01:27:09.639559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.639489 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b802cb-c008-4902-b352-fc95ca96d5a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b802cb-c008-4902-b352-fc95ca96d5a5-plugins-config-volume\") pod \"09b802cb-c008-4902-b352-fc95ca96d5a5\" (UID: \"09b802cb-c008-4902-b352-fc95ca96d5a5\") " Oct 30 01:27:09.639944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.639914 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e8119b-10cc-4552-ae70-5b621711f586-plugins-config-volume" pod "46e8119b-10cc-4552-ae70-5b621711f586" (UID: "46e8119b-10cc-4552-ae70-5b621711f586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:09.639944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.639933 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b802cb-c008-4902-b352-fc95ca96d5a5-plugins-config-volume" pod "09b802cb-c008-4902-b352-fc95ca96d5a5" (UID: "09b802cb-c008-4902-b352-fc95ca96d5a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:09.642153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.642130 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e8119b-10cc-4552-ae70-5b621711f586-kube-api-access-w6wfb" pod "46e8119b-10cc-4552-ae70-5b621711f586" (UID: "46e8119b-10cc-4552-ae70-5b621711f586"). InnerVolumeSpecName "kube-api-access-w6wfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:09.642246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.642228 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b802cb-c008-4902-b352-fc95ca96d5a5-kube-api-access-f64wz" pod "09b802cb-c008-4902-b352-fc95ca96d5a5" (UID: "09b802cb-c008-4902-b352-fc95ca96d5a5"). InnerVolumeSpecName "kube-api-access-f64wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:09.740608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.740570 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b802cb-c008-4902-b352-fc95ca96d5a5-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:09.740608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.740598 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6wfb\" (UniqueName: \"kubernetes.io/projected/46e8119b-10cc-4552-ae70-5b621711f586-kube-api-access-w6wfb\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:09.740608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.740608 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f64wz\" (UniqueName: \"kubernetes.io/projected/09b802cb-c008-4902-b352-fc95ca96d5a5-kube-api-access-f64wz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:09.740861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.740618 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e8119b-10cc-4552-ae70-5b621711f586-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:09.841259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.841163 4905 generic.go:356] "Generic (PLEG): container finished" podID="46e8119b-10cc-4552-ae70-5b621711f586" containerID="018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225" exitCode=0 Oct 30 01:27:09.841259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.841206 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" Oct 30 01:27:09.841549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.841252 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" event={"ID":"46e8119b-10cc-4552-ae70-5b621711f586","Type":"ContainerDied","Data":"018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225"} Oct 30 01:27:09.841549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.841334 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl" event={"ID":"46e8119b-10cc-4552-ae70-5b621711f586","Type":"ContainerDied","Data":"037ca05607df9ca843398fd1679856f243685881f45a28ba3ea4e98f8eced614"} Oct 30 01:27:09.841549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.841363 4905 scope.go:122] "RemoveContainer" containerID="018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225" Oct 30 01:27:09.842942 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.842885 4905 generic.go:356] "Generic (PLEG): container finished" podID="09b802cb-c008-4902-b352-fc95ca96d5a5" containerID="9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f" exitCode=0 Oct 30 01:27:09.843022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.842942 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" Oct 30 01:27:09.843022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.842974 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" event={"ID":"09b802cb-c008-4902-b352-fc95ca96d5a5","Type":"ContainerDied","Data":"9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f"} Oct 30 01:27:09.843149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.843037 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9" event={"ID":"09b802cb-c008-4902-b352-fc95ca96d5a5","Type":"ContainerDied","Data":"17207606109c0e810ef939e3acec1051a3b2b9772d3558b6a80707eb1f317368"} Oct 30 01:27:09.855011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.854991 4905 scope.go:122] "RemoveContainer" containerID="018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225" Oct 30 01:27:09.855285 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:09.855263 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225\": container with ID starting with 018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225 not found: ID does not exist" containerID="018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225" Oct 30 01:27:09.855365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.855296 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225"} err="failed to get container status \"018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225\": rpc error: code = NotFound desc = could not find container \"018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225\": container with ID starting with 018bfe83a3035cb35dbc23e8ae80e57e26d0b7f57a820a8a3ef50c5aaee6a225 not found: ID does not exist" Oct 30 01:27:09.855365 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.855320 4905 scope.go:122] "RemoveContainer" containerID="9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f" Oct 30 01:27:09.865348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.865329 4905 scope.go:122] "RemoveContainer" containerID="9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f" Oct 30 01:27:09.865688 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:09.865665 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f\": container with ID starting with 9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f not found: ID does not exist" containerID="9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f" Oct 30 01:27:09.865779 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.865697 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f"} err="failed to get container status \"9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f\": rpc error: code = NotFound desc = could not find container \"9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f\": container with ID starting with 9ef61c550ce9a4664f9406f68701f18cf3b5208873c7b360dd7b0fca99659f5f not found: ID does not exist" Oct 30 01:27:09.896823 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.896784 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl"] Oct 30 01:27:09.900321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.900294 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ad00a0-5c7b6b449f-qvgsl"] Oct 30 01:27:09.934972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.934939 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9"] Oct 30 01:27:09.937292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:09.937272 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ad00a0-c45c56f9b-ttgj9"] Oct 30 01:27:11.191730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:11.191694 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b802cb-c008-4902-b352-fc95ca96d5a5" path="/var/lib/kubelet/pods/09b802cb-c008-4902-b352-fc95ca96d5a5/volumes" Oct 30 01:27:11.192142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:11.192107 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e8119b-10cc-4552-ae70-5b621711f586" path="/var/lib/kubelet/pods/46e8119b-10cc-4552-ae70-5b621711f586/volumes" Oct 30 01:27:12.414007 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:12.413975 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:27:13.240617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.240584 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb"] Oct 30 01:27:13.241223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.241182 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" containerID="cri-o://5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75" gracePeriod=30 Oct 30 01:27:13.812514 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.812475 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb"] Oct 30 01:27:13.813068 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813052 4905 cpu_manager.go:414] "containerMap: removing container" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" Oct 30 01:27:13.813110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813070 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" Oct 30 01:27:13.813110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813080 4905 cpu_manager.go:414] "containerMap: removing container" podUID="736ddaa0-2e2d-42c8-8277-0f9ffe3da974" containerName="payload-pre-processing" Oct 30 01:27:13.813110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813086 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736ddaa0-2e2d-42c8-8277-0f9ffe3da974" containerName="payload-pre-processing" Oct 30 01:27:13.813110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813095 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ada8efa9-0769-4886-9d93-14b541f44f69" containerName="maas-api" Oct 30 01:27:13.813110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813101 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada8efa9-0769-4886-9d93-14b541f44f69" containerName="maas-api" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813121 4905 cpu_manager.go:414] "containerMap: removing container" podUID="46e8119b-10cc-4552-ae70-5b621711f586" containerName="payload-pre-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813126 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e8119b-10cc-4552-ae70-5b621711f586" containerName="payload-pre-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813135 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1df68de3-3599-4f4c-9e25-7e85f942467d" containerName="payload-pre-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813140 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df68de3-3599-4f4c-9e25-7e85f942467d" containerName="payload-pre-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813152 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813156 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813168 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813172 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813180 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7e1f673b-e586-44f2-acc4-d67c2fe94bef" containerName="payload-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813184 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1f673b-e586-44f2-acc4-d67c2fe94bef" containerName="payload-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813190 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1e90d226-6961-4cd5-8132-77c006205f67" containerName="payload-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813194 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e90d226-6961-4cd5-8132-77c006205f67" containerName="payload-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813200 4905 cpu_manager.go:414] "containerMap: removing container" podUID="09b802cb-c008-4902-b352-fc95ca96d5a5" containerName="payload-processing" Oct 30 01:27:13.813268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813204 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b802cb-c008-4902-b352-fc95ca96d5a5" containerName="payload-processing" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813281 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813290 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e381e2-5b6b-49af-8fe2-5ba37a189955" containerName="istio-proxy" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813297 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1f673b-e586-44f2-acc4-d67c2fe94bef" containerName="payload-processing" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813303 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e90d226-6961-4cd5-8132-77c006205f67" containerName="payload-processing" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813308 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b802cb-c008-4902-b352-fc95ca96d5a5" containerName="payload-processing" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813316 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b712d86-c47a-41a9-9a25-1ae45c0ba963" containerName="istio-proxy" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813321 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="736ddaa0-2e2d-42c8-8277-0f9ffe3da974" containerName="payload-pre-processing" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813328 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada8efa9-0769-4886-9d93-14b541f44f69" containerName="maas-api" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813335 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e8119b-10cc-4552-ae70-5b621711f586" containerName="payload-pre-processing" Oct 30 01:27:13.813744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.813340 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df68de3-3599-4f4c-9e25-7e85f942467d" containerName="payload-pre-processing" Oct 30 01:27:13.818203 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.818183 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.836027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.835488 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:13.836027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.835678 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:27:13.836387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.836364 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb"] Oct 30 01:27:13.890395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.890363 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.890636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.890427 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.890636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.890606 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjxj\" (UniqueName: \"kubernetes.io/projected/23d206ae-a148-4005-8bba-d4c3b7eb75ba-kube-api-access-qsjxj\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.991481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.991442 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.991725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.991504 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.991725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.991623 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjxj\" (UniqueName: \"kubernetes.io/projected/23d206ae-a148-4005-8bba-d4c3b7eb75ba-kube-api-access-qsjxj\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.992749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.992351 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:13.994336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:13.994312 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:14.013185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:14.013153 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjxj\" (UniqueName: \"kubernetes.io/projected/23d206ae-a148-4005-8bba-d4c3b7eb75ba-kube-api-access-qsjxj\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:14.133589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:14.133488 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:14.440263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:14.440219 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb"] Oct 30 01:27:14.905554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:14.905485 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" event={"ID":"23d206ae-a148-4005-8bba-d4c3b7eb75ba","Type":"ContainerStarted","Data":"bf415f8d07d2c62636b59cc58818708768edcd876079f448428522e0fc2a2691"} Oct 30 01:27:14.905554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:14.905556 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" event={"ID":"23d206ae-a148-4005-8bba-d4c3b7eb75ba","Type":"ContainerStarted","Data":"ba472b1ec48954c4b769404aac5693b5da304af05b212e35c039c3b54d4d029a"} Oct 30 01:27:14.946243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:14.946172 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" podStartSLOduration=1.9461538630000002 podStartE2EDuration="1.946153863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:14.945101431 +0000 UTC m=+2919.888274625" watchObservedRunningTime="2026-10-30 01:27:14.946153863 +0000 UTC m=+2919.889327056" Oct 30 01:27:15.913701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:15.913663 4905 generic.go:356] "Generic (PLEG): container finished" podID="23d206ae-a148-4005-8bba-d4c3b7eb75ba" containerID="bf415f8d07d2c62636b59cc58818708768edcd876079f448428522e0fc2a2691" exitCode=0 Oct 30 01:27:15.914126 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:15.913739 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" event={"ID":"23d206ae-a148-4005-8bba-d4c3b7eb75ba","Type":"ContainerDied","Data":"bf415f8d07d2c62636b59cc58818708768edcd876079f448428522e0fc2a2691"} Oct 30 01:27:17.090134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.090109 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:17.128439 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.128410 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-openshift-ca\") pod \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " Oct 30 01:27:17.128643 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.128598 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d206ae-a148-4005-8bba-d4c3b7eb75ba-kube-api-access-qsjxj\" (UniqueName: \"kubernetes.io/projected/23d206ae-a148-4005-8bba-d4c3b7eb75ba-kube-api-access-qsjxj\") pod \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " Oct 30 01:27:17.128734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.128643 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-xks-ca\") pod \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\" (UID: \"23d206ae-a148-4005-8bba-d4c3b7eb75ba\") " Oct 30 01:27:17.128850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.128829 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-openshift-ca" pod "23d206ae-a148-4005-8bba-d4c3b7eb75ba" (UID: "23d206ae-a148-4005-8bba-d4c3b7eb75ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:17.129104 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.129087 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:17.130881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.130856 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-xks-ca" pod "23d206ae-a148-4005-8bba-d4c3b7eb75ba" (UID: "23d206ae-a148-4005-8bba-d4c3b7eb75ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:17.131044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.131027 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d206ae-a148-4005-8bba-d4c3b7eb75ba-kube-api-access-qsjxj" pod "23d206ae-a148-4005-8bba-d4c3b7eb75ba" (UID: "23d206ae-a148-4005-8bba-d4c3b7eb75ba"). InnerVolumeSpecName "kube-api-access-qsjxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:17.230245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.230215 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjxj\" (UniqueName: \"kubernetes.io/projected/23d206ae-a148-4005-8bba-d4c3b7eb75ba-kube-api-access-qsjxj\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:17.230245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.230245 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d206ae-a148-4005-8bba-d4c3b7eb75ba-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:17.927045 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.927011 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" Oct 30 01:27:17.927259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.927062 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb" event={"ID":"23d206ae-a148-4005-8bba-d4c3b7eb75ba","Type":"ContainerDied","Data":"ba472b1ec48954c4b769404aac5693b5da304af05b212e35c039c3b54d4d029a"} Oct 30 01:27:17.927259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:17.927121 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba472b1ec48954c4b769404aac5693b5da304af05b212e35c039c3b54d4d029a" Oct 30 01:27:18.062107 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.062062 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" containerID="cri-o://ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861" gracePeriod=45 Oct 30 01:27:18.387673 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.387639 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:27:18.458557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.457982 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-home\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-home\") pod \"dff3d451-bc75-41a9-8048-26cbb03cab06\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " Oct 30 01:27:18.458557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458117 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff3d451-bc75-41a9-8048-26cbb03cab06-kube-api-access-f6rjh\" (UniqueName: \"kubernetes.io/projected/dff3d451-bc75-41a9-8048-26cbb03cab06-kube-api-access-f6rjh\") pod \"dff3d451-bc75-41a9-8048-26cbb03cab06\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " Oct 30 01:27:18.458557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458177 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-dshm\") pod \"dff3d451-bc75-41a9-8048-26cbb03cab06\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " Oct 30 01:27:18.458557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458196 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dff3d451-bc75-41a9-8048-26cbb03cab06-tls-certs\" (UniqueName: \"kubernetes.io/secret/dff3d451-bc75-41a9-8048-26cbb03cab06-tls-certs\") pod \"dff3d451-bc75-41a9-8048-26cbb03cab06\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " Oct 30 01:27:18.458557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458255 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-tmp-dir\") pod \"dff3d451-bc75-41a9-8048-26cbb03cab06\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " Oct 30 01:27:18.458557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458291 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-model-cache\") pod \"dff3d451-bc75-41a9-8048-26cbb03cab06\" (UID: \"dff3d451-bc75-41a9-8048-26cbb03cab06\") " Oct 30 01:27:18.458557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458293 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-home" pod "dff3d451-bc75-41a9-8048-26cbb03cab06" (UID: "dff3d451-bc75-41a9-8048-26cbb03cab06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.459113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458631 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.459113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.458845 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-model-cache" pod "dff3d451-bc75-41a9-8048-26cbb03cab06" (UID: "dff3d451-bc75-41a9-8048-26cbb03cab06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.459113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.459024 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-tmp-dir" pod "dff3d451-bc75-41a9-8048-26cbb03cab06" (UID: "dff3d451-bc75-41a9-8048-26cbb03cab06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.462965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.462919 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff3d451-bc75-41a9-8048-26cbb03cab06-kube-api-access-f6rjh" pod "dff3d451-bc75-41a9-8048-26cbb03cab06" (UID: "dff3d451-bc75-41a9-8048-26cbb03cab06"). InnerVolumeSpecName "kube-api-access-f6rjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:18.463157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.463136 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff3d451-bc75-41a9-8048-26cbb03cab06-tls-certs" pod "dff3d451-bc75-41a9-8048-26cbb03cab06" (UID: "dff3d451-bc75-41a9-8048-26cbb03cab06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:18.463268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.463255 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-dshm" pod "dff3d451-bc75-41a9-8048-26cbb03cab06" (UID: "dff3d451-bc75-41a9-8048-26cbb03cab06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.502387 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.502365 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:27:18.559254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559221 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220ad5d5-3373-4eb3-a4c2-d9f068297389-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220ad5d5-3373-4eb3-a4c2-d9f068297389-istiod-ca-cert\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.559672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559281 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-certs\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.559672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559347 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-credential-socket\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.559672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559478 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-data\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.559672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559557 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-kube-api-access-dd2f5\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-kube-api-access-dd2f5\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.559672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559609 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-podinfo\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.559672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559629 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-token\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-token\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.559672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559652 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-socket\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559695 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-envoy\") pod \"220ad5d5-3373-4eb3-a4c2-d9f068297389\" (UID: \"220ad5d5-3373-4eb3-a4c2-d9f068297389\") " Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559713 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-certs" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.559761 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220ad5d5-3373-4eb3-a4c2-d9f068297389-istiod-ca-cert" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560051 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560070 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560083 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rjh\" (UniqueName: \"kubernetes.io/projected/dff3d451-bc75-41a9-8048-26cbb03cab06-kube-api-access-f6rjh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560096 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560109 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff3d451-bc75-41a9-8048-26cbb03cab06-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560125 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220ad5d5-3373-4eb3-a4c2-d9f068297389-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.560148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560136 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff3d451-bc75-41a9-8048-26cbb03cab06-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.560968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560908 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-credential-socket" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.560968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.560932 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-socket" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.561236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.561214 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-data" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.562698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.562677 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-kube-api-access-dd2f5" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "kube-api-access-dd2f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:18.562815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.562717 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-token" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:18.563848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.563004 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-envoy" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:18.563848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.563634 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-podinfo" pod "220ad5d5-3373-4eb3-a4c2-d9f068297389" (UID: "220ad5d5-3373-4eb3-a4c2-d9f068297389"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:27:18.660911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.660882 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.660911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.660907 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.660911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.660916 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2f5\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-kube-api-access-dd2f5\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.661157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.660925 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.661157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.660935 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.661157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.660944 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.661157 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.660952 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220ad5d5-3373-4eb3-a4c2-d9f068297389-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:18.936148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.936071 4905 generic.go:356] "Generic (PLEG): container finished" podID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerID="ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861" exitCode=0 Oct 30 01:27:18.936148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.936131 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" Oct 30 01:27:18.936371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.936154 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" event={"ID":"dff3d451-bc75-41a9-8048-26cbb03cab06","Type":"ContainerDied","Data":"ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861"} Oct 30 01:27:18.936371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.936196 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz" event={"ID":"dff3d451-bc75-41a9-8048-26cbb03cab06","Type":"ContainerDied","Data":"ce423bd09703343e69cf83c562396ad4ca7765863a9bba3af94944cb587013d0"} Oct 30 01:27:18.936371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.936219 4905 scope.go:122] "RemoveContainer" containerID="ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861" Oct 30 01:27:18.938561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.937959 4905 generic.go:356] "Generic (PLEG): container finished" podID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerID="5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75" exitCode=0 Oct 30 01:27:18.938561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.938024 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" Oct 30 01:27:18.938561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.938063 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" event={"ID":"220ad5d5-3373-4eb3-a4c2-d9f068297389","Type":"ContainerDied","Data":"5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75"} Oct 30 01:27:18.938561 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.938118 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb" event={"ID":"220ad5d5-3373-4eb3-a4c2-d9f068297389","Type":"ContainerDied","Data":"3ca2b5a3377a5615e81d1c8d15f42a4194b7ffa182f665a6e4653c47abf206e5"} Oct 30 01:27:18.946254 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.946235 4905 scope.go:122] "RemoveContainer" containerID="ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861" Oct 30 01:27:18.946565 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:18.946540 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861\": container with ID starting with ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861 not found: ID does not exist" containerID="ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861" Oct 30 01:27:18.946663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.946570 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861"} err="failed to get container status \"ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861\": rpc error: code = NotFound desc = could not find container \"ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861\": container with ID starting with ec52c940055b0bce8a107b2fde86d1135fb583a4771c4264fa3698fdef23d861 not found: ID does not exist" Oct 30 01:27:18.946663 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.946588 4905 scope.go:122] "RemoveContainer" containerID="5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75" Oct 30 01:27:18.959432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.959416 4905 scope.go:122] "RemoveContainer" containerID="f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd" Oct 30 01:27:18.979050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.979025 4905 scope.go:122] "RemoveContainer" containerID="5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75" Oct 30 01:27:18.979387 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:18.979362 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75\": container with ID starting with 5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75 not found: ID does not exist" containerID="5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75" Oct 30 01:27:18.979571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.979399 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75"} err="failed to get container status \"5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75\": rpc error: code = NotFound desc = could not find container \"5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75\": container with ID starting with 5416b0c35cc028c4d4f7befb8e9288340afea12a05668ee3afe6f19bc3e0df75 not found: ID does not exist" Oct 30 01:27:18.979651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.979577 4905 scope.go:122] "RemoveContainer" containerID="f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd" Oct 30 01:27:18.979946 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:18.979925 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd\": container with ID starting with f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd not found: ID does not exist" containerID="f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd" Oct 30 01:27:18.980043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.979953 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd"} err="failed to get container status \"f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd\": rpc error: code = NotFound desc = could not find container \"f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd\": container with ID starting with f9974c8585883a640686f974787604972182f4c073c2ff01db3a4200698525bd not found: ID does not exist" Oct 30 01:27:18.993895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:18.993872 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb"] Oct 30 01:27:19.014310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:19.014279 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0519f6-openshift-default-66cbb54c77-nbxgb"] Oct 30 01:27:19.053171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:19.053136 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz"] Oct 30 01:27:19.058228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:19.058201 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/premium-w1-8cf33a-kserve-5df586d94c-gl2dz"] Oct 30 01:27:19.192202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:19.192128 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" path="/var/lib/kubelet/pods/220ad5d5-3373-4eb3-a4c2-d9f068297389/volumes" Oct 30 01:27:19.192592 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:19.192573 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" path="/var/lib/kubelet/pods/dff3d451-bc75-41a9-8048-26cbb03cab06/volumes" Oct 30 01:27:24.725651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.723309 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2"] Oct 30 01:27:24.725651 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.723677 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" podUID="fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" containerName="maas-api" containerID="cri-o://a52a300f50a54425203d4110f06e2f1256e630a9f0740f3c7292a80a8a3755bd" gracePeriod=30 Oct 30 01:27:24.861158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.861127 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww"] Oct 30 01:27:24.861824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.861359 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" podUID="ec3c12cb-c978-4680-8e6e-c230369fbf13" containerName="payload-processing" containerID="cri-o://4773058a0338d81337e2d229c6a917a1084d84578938eb576574cd5aeda7c528" gracePeriod=30 Oct 30 01:27:24.879075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.879037 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b"] Oct 30 01:27:24.879303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.879277 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" podUID="52ec8470-e62a-4d21-b66f-debbcfbca51f" containerName="payload-pre-processing" containerID="cri-o://5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864" gracePeriod=30 Oct 30 01:27:24.985899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.985672 4905 generic.go:356] "Generic (PLEG): container finished" podID="ec3c12cb-c978-4680-8e6e-c230369fbf13" containerID="4773058a0338d81337e2d229c6a917a1084d84578938eb576574cd5aeda7c528" exitCode=0 Oct 30 01:27:24.985899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.985739 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" event={"ID":"ec3c12cb-c978-4680-8e6e-c230369fbf13","Type":"ContainerDied","Data":"4773058a0338d81337e2d229c6a917a1084d84578938eb576574cd5aeda7c528"} Oct 30 01:27:24.988342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.988317 4905 generic.go:356] "Generic (PLEG): container finished" podID="fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" containerID="a52a300f50a54425203d4110f06e2f1256e630a9f0740f3c7292a80a8a3755bd" exitCode=0 Oct 30 01:27:24.988505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:24.988363 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" event={"ID":"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5","Type":"ContainerDied","Data":"a52a300f50a54425203d4110f06e2f1256e630a9f0740f3c7292a80a8a3755bd"} Oct 30 01:27:25.099196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.099164 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:27:25.258716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.255590 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-openshift-service-ca\") pod \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " Oct 30 01:27:25.258716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.255966 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-openshift-service-ca" pod "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" (UID: "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:25.258716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.256084 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-kube-api-access-vvfhn\" (UniqueName: \"kubernetes.io/projected/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-kube-api-access-vvfhn\") pod \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " Oct 30 01:27:25.258716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.256176 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-metrics-certs\") pod \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " Oct 30 01:27:25.258716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.256272 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-maas-api-tls\") pod \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\" (UID: \"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5\") " Oct 30 01:27:25.258716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.256780 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.273551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.271489 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-kube-api-access-vvfhn" pod "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" (UID: "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5"). InnerVolumeSpecName "kube-api-access-vvfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:25.273551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.271799 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-maas-api-tls" pod "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" (UID: "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:25.273551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.272496 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-metrics-certs" pod "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" (UID: "fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:25.285795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.285768 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:27:25.334837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.334809 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:27:25.358702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.358671 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvfhn\" (UniqueName: \"kubernetes.io/projected/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-kube-api-access-vvfhn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.358702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.358706 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.358962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.358722 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.439140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.439102 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb"] Oct 30 01:27:25.441399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.441374 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c37c9-n8nmb"] Oct 30 01:27:25.459590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.459557 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ec8470-e62a-4d21-b66f-debbcfbca51f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec8470-e62a-4d21-b66f-debbcfbca51f-plugins-config-volume\") pod \"52ec8470-e62a-4d21-b66f-debbcfbca51f\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " Oct 30 01:27:25.459728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.459662 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ec8470-e62a-4d21-b66f-debbcfbca51f-kube-api-access-rz8g7\" (UniqueName: \"kubernetes.io/projected/52ec8470-e62a-4d21-b66f-debbcfbca51f-kube-api-access-rz8g7\") pod \"52ec8470-e62a-4d21-b66f-debbcfbca51f\" (UID: \"52ec8470-e62a-4d21-b66f-debbcfbca51f\") " Oct 30 01:27:25.459801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.459768 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3c12cb-c978-4680-8e6e-c230369fbf13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3c12cb-c978-4680-8e6e-c230369fbf13-plugins-config-volume\") pod \"ec3c12cb-c978-4680-8e6e-c230369fbf13\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " Oct 30 01:27:25.459864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.459804 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3c12cb-c978-4680-8e6e-c230369fbf13-kube-api-access-9cq6c\" (UniqueName: \"kubernetes.io/projected/ec3c12cb-c978-4680-8e6e-c230369fbf13-kube-api-access-9cq6c\") pod \"ec3c12cb-c978-4680-8e6e-c230369fbf13\" (UID: \"ec3c12cb-c978-4680-8e6e-c230369fbf13\") " Oct 30 01:27:25.460096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.460064 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ec8470-e62a-4d21-b66f-debbcfbca51f-plugins-config-volume" pod "52ec8470-e62a-4d21-b66f-debbcfbca51f" (UID: "52ec8470-e62a-4d21-b66f-debbcfbca51f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:25.460261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.460141 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3c12cb-c978-4680-8e6e-c230369fbf13-plugins-config-volume" pod "ec3c12cb-c978-4680-8e6e-c230369fbf13" (UID: "ec3c12cb-c978-4680-8e6e-c230369fbf13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:25.460261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.460224 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3c12cb-c978-4680-8e6e-c230369fbf13-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.460261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.460245 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec8470-e62a-4d21-b66f-debbcfbca51f-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.462076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.462055 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ec8470-e62a-4d21-b66f-debbcfbca51f-kube-api-access-rz8g7" pod "52ec8470-e62a-4d21-b66f-debbcfbca51f" (UID: "52ec8470-e62a-4d21-b66f-debbcfbca51f"). InnerVolumeSpecName "kube-api-access-rz8g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:25.462311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.462289 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3c12cb-c978-4680-8e6e-c230369fbf13-kube-api-access-9cq6c" pod "ec3c12cb-c978-4680-8e6e-c230369fbf13" (UID: "ec3c12cb-c978-4680-8e6e-c230369fbf13"). InnerVolumeSpecName "kube-api-access-9cq6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:25.561596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.561496 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz8g7\" (UniqueName: \"kubernetes.io/projected/52ec8470-e62a-4d21-b66f-debbcfbca51f-kube-api-access-rz8g7\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.561596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.561545 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cq6c\" (UniqueName: \"kubernetes.io/projected/ec3c12cb-c978-4680-8e6e-c230369fbf13-kube-api-access-9cq6c\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.997044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.997011 4905 generic.go:356] "Generic (PLEG): container finished" podID="52ec8470-e62a-4d21-b66f-debbcfbca51f" containerID="5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864" exitCode=0 Oct 30 01:27:25.997582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.997074 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" Oct 30 01:27:25.997582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.997094 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" event={"ID":"52ec8470-e62a-4d21-b66f-debbcfbca51f","Type":"ContainerDied","Data":"5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864"} Oct 30 01:27:25.997582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.997152 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b" event={"ID":"52ec8470-e62a-4d21-b66f-debbcfbca51f","Type":"ContainerDied","Data":"7bde3e60f789a73111c46ce552994657ee50c9d6facb7b1fc8e7473a3c985710"} Oct 30 01:27:25.997582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:25.997175 4905 scope.go:122] "RemoveContainer" containerID="5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864" Oct 30 01:27:26.004015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.003969 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" Oct 30 01:27:26.004394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.004365 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2" event={"ID":"fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5","Type":"ContainerDied","Data":"c8adb174bd051b7f4e6b5c7dfc659ba71f98c1b59b69ef41c838bf236c15bb7b"} Oct 30 01:27:26.008477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.006547 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" event={"ID":"ec3c12cb-c978-4680-8e6e-c230369fbf13","Type":"ContainerDied","Data":"eab1c3b05b809150e81ebd9b4923555cd33926f32edbe460212db6009781cf6e"} Oct 30 01:27:26.008477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.006567 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww" Oct 30 01:27:26.013843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.013823 4905 scope.go:122] "RemoveContainer" containerID="5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864" Oct 30 01:27:26.014272 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:26.014246 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864\": container with ID starting with 5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864 not found: ID does not exist" containerID="5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864" Oct 30 01:27:26.014364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.014284 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864"} err="failed to get container status \"5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864\": rpc error: code = NotFound desc = could not find container \"5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864\": container with ID starting with 5a5452c6b82f450b5b97b164e0f73611a90b04f1d31b0906f649a4f47fbbb864 not found: ID does not exist" Oct 30 01:27:26.014364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.014302 4905 scope.go:122] "RemoveContainer" containerID="a52a300f50a54425203d4110f06e2f1256e630a9f0740f3c7292a80a8a3755bd" Oct 30 01:27:26.023740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.023722 4905 scope.go:122] "RemoveContainer" containerID="4773058a0338d81337e2d229c6a917a1084d84578938eb576574cd5aeda7c528" Oct 30 01:27:26.068208 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.068179 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b"] Oct 30 01:27:26.081854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.081824 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c37c9-685b5694f9-kz69b"] Oct 30 01:27:26.127875 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.127847 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww"] Oct 30 01:27:26.131084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.131057 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c37c9-6b8f99dbf4-56nww"] Oct 30 01:27:26.168880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.168853 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2"] Oct 30 01:27:26.172669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:26.172640 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c37c9-5f59c8c99-nrcq2"] Oct 30 01:27:27.191764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:27.191723 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d206ae-a148-4005-8bba-d4c3b7eb75ba" path="/var/lib/kubelet/pods/23d206ae-a148-4005-8bba-d4c3b7eb75ba/volumes" Oct 30 01:27:27.192186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:27.192171 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ec8470-e62a-4d21-b66f-debbcfbca51f" path="/var/lib/kubelet/pods/52ec8470-e62a-4d21-b66f-debbcfbca51f/volumes" Oct 30 01:27:27.192575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:27.192558 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3c12cb-c978-4680-8e6e-c230369fbf13" path="/var/lib/kubelet/pods/ec3c12cb-c978-4680-8e6e-c230369fbf13/volumes" Oct 30 01:27:27.193289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:27.193274 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" path="/var/lib/kubelet/pods/fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5/volumes" Oct 30 01:27:32.020786 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:32.020756 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b"] Oct 30 01:27:32.021227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:32.020967 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" containerID="cri-o://ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b" gracePeriod=30 Oct 30 01:27:36.364848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:36.359624 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh"] Oct 30 01:27:36.364848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:36.359933 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" containerID="cri-o://9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc" gracePeriod=30 Oct 30 01:27:37.294307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.294243 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:27:37.309999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.309960 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-envoy\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310082 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-data\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310126 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72332109-52dc-4c45-9507-a45b122b54d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72332109-52dc-4c45-9507-a45b122b54d0-istio-podinfo\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310190 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72332109-52dc-4c45-9507-a45b122b54d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72332109-52dc-4c45-9507-a45b122b54d0-istiod-ca-cert\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310288 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-kube-api-access-v6vh4\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-kube-api-access-v6vh4\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310383 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310326 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-data" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:37.310383 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310339 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-certs\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310408 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-credential-socket\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310473 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-istio-token\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-istio-token\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310544 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-socket\") pod \"72332109-52dc-4c45-9507-a45b122b54d0\" (UID: \"72332109-52dc-4c45-9507-a45b122b54d0\") " Oct 30 01:27:37.310705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310643 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-certs" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:37.310705 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310661 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72332109-52dc-4c45-9507-a45b122b54d0-istiod-ca-cert" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:37.311004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310950 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-socket" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:37.311118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.311017 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-credential-socket" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:37.311118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.310973 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.311224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.311174 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72332109-52dc-4c45-9507-a45b122b54d0-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.311224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.311193 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.312944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.312915 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72332109-52dc-4c45-9507-a45b122b54d0-istio-podinfo" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:27:37.312944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.312925 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-envoy" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:37.313131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.313113 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-istio-token" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:37.313199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.313174 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-kube-api-access-v6vh4" pod "72332109-52dc-4c45-9507-a45b122b54d0" (UID: "72332109-52dc-4c45-9507-a45b122b54d0"). InnerVolumeSpecName "kube-api-access-v6vh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:37.412347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.412312 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.412347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.412345 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.412347 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.412354 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72332109-52dc-4c45-9507-a45b122b54d0-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.412844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.412363 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6vh4\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-kube-api-access-v6vh4\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.412844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.412373 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72332109-52dc-4c45-9507-a45b122b54d0-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:37.412844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:37.412382 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72332109-52dc-4c45-9507-a45b122b54d0-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:38.096482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.096443 4905 generic.go:356] "Generic (PLEG): container finished" podID="72332109-52dc-4c45-9507-a45b122b54d0" containerID="ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b" exitCode=0 Oct 30 01:27:38.096723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.096571 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" event={"ID":"72332109-52dc-4c45-9507-a45b122b54d0","Type":"ContainerDied","Data":"ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b"} Oct 30 01:27:38.096723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.096593 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" Oct 30 01:27:38.096723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.096648 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b" event={"ID":"72332109-52dc-4c45-9507-a45b122b54d0","Type":"ContainerDied","Data":"422c9c542782d39050a18e8e1bc4ad07f3d5eb8d182465e3d8a811d1a2eddc8e"} Oct 30 01:27:38.096723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.096677 4905 scope.go:122] "RemoveContainer" containerID="ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b" Oct 30 01:27:38.110091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.110068 4905 scope.go:122] "RemoveContainer" containerID="2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d" Oct 30 01:27:38.119621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.119596 4905 scope.go:122] "RemoveContainer" containerID="ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b" Oct 30 01:27:38.119882 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:38.119863 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b\": container with ID starting with ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b not found: ID does not exist" containerID="ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b" Oct 30 01:27:38.119971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.119895 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b"} err="failed to get container status \"ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b\": rpc error: code = NotFound desc = could not find container \"ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b\": container with ID starting with ec79ee2c0c0af3aaddd58c17b7bf3b694ebe417ab90a154d3d99124769eb7c7b not found: ID does not exist" Oct 30 01:27:38.119971 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.119918 4905 scope.go:122] "RemoveContainer" containerID="2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d" Oct 30 01:27:38.120193 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:38.120172 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d\": container with ID starting with 2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d not found: ID does not exist" containerID="2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d" Oct 30 01:27:38.120237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.120200 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d"} err="failed to get container status \"2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d\": rpc error: code = NotFound desc = could not find container \"2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d\": container with ID starting with 2e9dcf303e70345b116f8c96236ed87dc9bc7dbec0b03943bd8299be7412323d not found: ID does not exist" Oct 30 01:27:38.165317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.165283 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b"] Oct 30 01:27:38.178819 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:38.178791 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c37c9-openshift-default-69bd9dd6f5-sb57b"] Oct 30 01:27:39.190893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:39.190858 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72332109-52dc-4c45-9507-a45b122b54d0" path="/var/lib/kubelet/pods/72332109-52dc-4c45-9507-a45b122b54d0/volumes" Oct 30 01:27:41.643881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.643796 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 01:27:41.763680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.763645 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-token\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.763680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.763693 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-kube-api-access-8snk9\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-kube-api-access-8snk9\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.763961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.763747 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istiod-ca-cert\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.763961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.763788 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-socket\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.763961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.763835 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-podinfo\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.763961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.763864 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-envoy\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.763961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.763909 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-data\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.764235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764018 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-credential-socket\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.764235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764099 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-certs\") pod \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\" (UID: \"151a86fd-2313-46ba-9470-7fc0c8f22d1b\") " Oct 30 01:27:41.764235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764174 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istiod-ca-cert" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:41.764451 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764298 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-data" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:41.764451 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764365 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-socket" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:41.764603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764475 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-credential-socket" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:41.764603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764505 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-certs" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:41.764714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764678 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.764714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764698 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.764826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764712 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.764826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764728 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.764826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.764740 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.767167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.767118 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-token" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:41.767167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.767147 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-podinfo" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:27:41.767368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.767272 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-envoy" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:41.768000 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.767972 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-kube-api-access-8snk9" pod "151a86fd-2313-46ba-9470-7fc0c8f22d1b" (UID: "151a86fd-2313-46ba-9470-7fc0c8f22d1b"). InnerVolumeSpecName "kube-api-access-8snk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:41.866316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.866276 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.866316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.866317 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8snk9\" (UniqueName: \"kubernetes.io/projected/151a86fd-2313-46ba-9470-7fc0c8f22d1b-kube-api-access-8snk9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.866626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.866334 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:41.866626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:41.866348 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a86fd-2313-46ba-9470-7fc0c8f22d1b-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:42.125357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.125320 4905 generic.go:356] "Generic (PLEG): container finished" podID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerID="9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc" exitCode=0 Oct 30 01:27:42.125602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.125376 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" event={"ID":"151a86fd-2313-46ba-9470-7fc0c8f22d1b","Type":"ContainerDied","Data":"9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc"} Oct 30 01:27:42.125602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.125381 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" Oct 30 01:27:42.125602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.125422 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh" event={"ID":"151a86fd-2313-46ba-9470-7fc0c8f22d1b","Type":"ContainerDied","Data":"695bd8c50c97d07d7d8e920f553b5cd05745d9695c6a671c94e3affc4916b383"} Oct 30 01:27:42.125602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.125449 4905 scope.go:122] "RemoveContainer" containerID="9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc" Oct 30 01:27:42.137682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.137658 4905 scope.go:122] "RemoveContainer" containerID="909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d" Oct 30 01:27:42.146464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.146442 4905 scope.go:122] "RemoveContainer" containerID="9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc" Oct 30 01:27:42.146751 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:42.146733 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc\": container with ID starting with 9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc not found: ID does not exist" containerID="9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc" Oct 30 01:27:42.146817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.146761 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc"} err="failed to get container status \"9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc\": rpc error: code = NotFound desc = could not find container \"9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc\": container with ID starting with 9310980e7dfe2d2bc2fedc9e662729550f0e15d39ae1a4a8c3708561f98865cc not found: ID does not exist" Oct 30 01:27:42.146817 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.146778 4905 scope.go:122] "RemoveContainer" containerID="909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d" Oct 30 01:27:42.147047 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:42.147029 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d\": container with ID starting with 909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d not found: ID does not exist" containerID="909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d" Oct 30 01:27:42.147089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.147050 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d"} err="failed to get container status \"909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d\": rpc error: code = NotFound desc = could not find container \"909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d\": container with ID starting with 909382e0dab898b457269ecb90a08651252eb0881f03c3b502b38f804e5e5f4d not found: ID does not exist" Oct 30 01:27:42.180232 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.180198 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh"] Oct 30 01:27:42.184359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:42.184338 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-vcrrh"] Oct 30 01:27:42.414485 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:42.414393 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:27:43.037836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:43.037800 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk"] Oct 30 01:27:43.072927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:43.072892 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk"] Oct 30 01:27:43.193508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:43.193478 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" path="/var/lib/kubelet/pods/151a86fd-2313-46ba-9470-7fc0c8f22d1b/volumes" Oct 30 01:27:47.664581 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:47.664509 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf"] Oct 30 01:27:47.665076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:47.665014 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" podUID="36ec974d-9690-40be-b7ae-a8839b79d6db" containerName="maas-api" containerID="cri-o://0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355" gracePeriod=30 Oct 30 01:27:47.962143 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:47.962107 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:27:48.038749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.038674 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" podUID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" containerName="payload-processing" containerID="cri-o://859296d7de1cec95e06d6ffe2d9ae43bbf40e0f246f8b600ab302f8a4ea7a55f" gracePeriod=25 Oct 30 01:27:48.042982 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.042957 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ec974d-9690-40be-b7ae-a8839b79d6db-kube-api-access-cd667\" (UniqueName: \"kubernetes.io/projected/36ec974d-9690-40be-b7ae-a8839b79d6db-kube-api-access-cd667\") pod \"36ec974d-9690-40be-b7ae-a8839b79d6db\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " Oct 30 01:27:48.043776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.043076 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ec974d-9690-40be-b7ae-a8839b79d6db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec974d-9690-40be-b7ae-a8839b79d6db-openshift-service-ca\") pod \"36ec974d-9690-40be-b7ae-a8839b79d6db\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " Oct 30 01:27:48.043776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.043131 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-maas-api-tls\") pod \"36ec974d-9690-40be-b7ae-a8839b79d6db\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " Oct 30 01:27:48.043776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.043188 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-metrics-certs\") pod \"36ec974d-9690-40be-b7ae-a8839b79d6db\" (UID: \"36ec974d-9690-40be-b7ae-a8839b79d6db\") " Oct 30 01:27:48.043776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.043593 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ec974d-9690-40be-b7ae-a8839b79d6db-openshift-service-ca" pod "36ec974d-9690-40be-b7ae-a8839b79d6db" (UID: "36ec974d-9690-40be-b7ae-a8839b79d6db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:48.051313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.051280 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-metrics-certs" pod "36ec974d-9690-40be-b7ae-a8839b79d6db" (UID: "36ec974d-9690-40be-b7ae-a8839b79d6db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:48.055432 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.055400 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-maas-api-tls" pod "36ec974d-9690-40be-b7ae-a8839b79d6db" (UID: "36ec974d-9690-40be-b7ae-a8839b79d6db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:48.086429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.074405 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" podUID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" containerName="payload-pre-processing" containerID="cri-o://a491d5fce012bb888087a39d0268a06647b0215677b33a696cf7396297882fba" gracePeriod=25 Oct 30 01:27:48.086429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.076027 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ec974d-9690-40be-b7ae-a8839b79d6db-kube-api-access-cd667" pod "36ec974d-9690-40be-b7ae-a8839b79d6db" (UID: "36ec974d-9690-40be-b7ae-a8839b79d6db"). InnerVolumeSpecName "kube-api-access-cd667". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:48.144611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.144578 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec974d-9690-40be-b7ae-a8839b79d6db-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.144611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.144604 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.144611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.144614 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ec974d-9690-40be-b7ae-a8839b79d6db-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.144611 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.144621 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd667\" (UniqueName: \"kubernetes.io/projected/36ec974d-9690-40be-b7ae-a8839b79d6db-kube-api-access-cd667\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.177144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.177113 4905 generic.go:356] "Generic (PLEG): container finished" podID="36ec974d-9690-40be-b7ae-a8839b79d6db" containerID="0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355" exitCode=0 Oct 30 01:27:48.177323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.177159 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" Oct 30 01:27:48.177323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.177202 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" event={"ID":"36ec974d-9690-40be-b7ae-a8839b79d6db","Type":"ContainerDied","Data":"0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355"} Oct 30 01:27:48.177323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.177259 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf" event={"ID":"36ec974d-9690-40be-b7ae-a8839b79d6db","Type":"ContainerDied","Data":"69b1e9532d4e01da0460ac1222193314a96c266f643eb56f7e5ef419f2973ffe"} Oct 30 01:27:48.177323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.177287 4905 scope.go:122] "RemoveContainer" containerID="0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355" Oct 30 01:27:48.187477 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.187458 4905 scope.go:122] "RemoveContainer" containerID="0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355" Oct 30 01:27:48.187787 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:27:48.187768 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355\": container with ID starting with 0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355 not found: ID does not exist" containerID="0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355" Oct 30 01:27:48.187843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.187796 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355"} err="failed to get container status \"0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355\": rpc error: code = NotFound desc = could not find container \"0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355\": container with ID starting with 0a89fd3a39fcf3427dcdfa522266f6d1d9ed62ba39004ad36596360b68d3c355 not found: ID does not exist" Oct 30 01:27:48.234777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.234735 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf"] Oct 30 01:27:48.237584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.237558 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-60f38f-c5f978dfd-s62zf"] Oct 30 01:27:48.708378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.708329 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" podUID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:48.708842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:48.708427 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" podUID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:49.189014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:49.188977 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ec974d-9690-40be-b7ae-a8839b79d6db" path="/var/lib/kubelet/pods/36ec974d-9690-40be-b7ae-a8839b79d6db/volumes" Oct 30 01:27:54.786309 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:54.786276 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq"] Oct 30 01:27:54.786843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:54.786559 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" containerID="cri-o://1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f" gracePeriod=30 Oct 30 01:27:54.798752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:54.798717 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27"] Oct 30 01:27:54.798993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:54.798960 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" containerID="cri-o://1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8" gracePeriod=30 Oct 30 01:27:54.800919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:54.800895 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6"] Oct 30 01:27:54.801112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:54.801071 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" containerID="cri-o://ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246" gracePeriod=30 Oct 30 01:27:55.376948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.376911 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x"] Oct 30 01:27:55.377391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377378 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" containerName="maas-api" Oct 30 01:27:55.377436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377392 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" containerName="maas-api" Oct 30 01:27:55.377436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377401 4905 cpu_manager.go:414] "containerMap: removing container" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" Oct 30 01:27:55.377436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377406 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" Oct 30 01:27:55.377436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377415 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ec3c12cb-c978-4680-8e6e-c230369fbf13" containerName="payload-processing" Oct 30 01:27:55.377436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377420 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3c12cb-c978-4680-8e6e-c230369fbf13" containerName="payload-processing" Oct 30 01:27:55.377436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377431 4905 cpu_manager.go:414] "containerMap: removing container" podUID="52ec8470-e62a-4d21-b66f-debbcfbca51f" containerName="payload-pre-processing" Oct 30 01:27:55.377436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377436 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ec8470-e62a-4d21-b66f-debbcfbca51f" containerName="payload-pre-processing" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377446 4905 cpu_manager.go:414] "containerMap: removing container" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377451 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377460 4905 cpu_manager.go:414] "containerMap: removing container" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377465 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377477 4905 cpu_manager.go:414] "containerMap: removing container" podUID="36ec974d-9690-40be-b7ae-a8839b79d6db" containerName="maas-api" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377481 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ec974d-9690-40be-b7ae-a8839b79d6db" containerName="maas-api" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377487 4905 cpu_manager.go:414] "containerMap: removing container" podUID="23d206ae-a148-4005-8bba-d4c3b7eb75ba" containerName="revoke-keys" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377492 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d206ae-a148-4005-8bba-d4c3b7eb75ba" containerName="revoke-keys" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377503 4905 cpu_manager.go:414] "containerMap: removing container" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377507 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377513 4905 cpu_manager.go:414] "containerMap: removing container" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377535 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377542 4905 cpu_manager.go:414] "containerMap: removing container" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377546 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377629 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ec8470-e62a-4d21-b66f-debbcfbca51f" containerName="payload-pre-processing" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377638 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377646 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377653 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ec974d-9690-40be-b7ae-a8839b79d6db" containerName="maas-api" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377659 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d206ae-a148-4005-8bba-d4c3b7eb75ba" containerName="revoke-keys" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377665 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377670 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377676 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="72332109-52dc-4c45-9507-a45b122b54d0" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377681 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff3d451-bc75-41a9-8048-26cbb03cab06" containerName="main" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377690 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbacdd4-1e3d-4c95-ad51-655a5bb8a8d5" containerName="maas-api" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377695 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="220ad5d5-3373-4eb3-a4c2-d9f068297389" containerName="istio-proxy" Oct 30 01:27:55.377734 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.377701 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3c12cb-c978-4680-8e6e-c230369fbf13" containerName="payload-processing" Oct 30 01:27:55.381108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.381092 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.401966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.401933 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x"] Oct 30 01:27:55.413562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.412185 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:27:55.413562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.412244 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:55.413562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.412295 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:27:55.413562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.412295 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:27:55.529307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.529258 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsts\" (UniqueName: \"kubernetes.io/projected/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-kube-api-access-vrsts\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.529458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.529338 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.529458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.529396 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.630457 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.630356 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsts\" (UniqueName: \"kubernetes.io/projected/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-kube-api-access-vrsts\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.630457 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.630418 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.630457 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.630454 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.631056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.631036 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.633038 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.633012 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.651271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.651241 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsts\" (UniqueName: \"kubernetes.io/projected/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-kube-api-access-vrsts\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.695111 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.695079 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:55.993487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:55.993455 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x"] Oct 30 01:27:55.994092 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:27:55.994068 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc20abd_dca9_4273_bd5f_6d89ba9032d8.slice/crio-cee8ae27aa4e40c5057aeeffc90a6c13a97511fa332935002c066686348368d2 WatchSource:0}: Error finding container cee8ae27aa4e40c5057aeeffc90a6c13a97511fa332935002c066686348368d2: Status 404 returned error can't find the container with id cee8ae27aa4e40c5057aeeffc90a6c13a97511fa332935002c066686348368d2 Oct 30 01:27:56.224266 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:56.224232 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" event={"ID":"7fc20abd-dca9-4273-bd5f-6d89ba9032d8","Type":"ContainerStarted","Data":"cee8ae27aa4e40c5057aeeffc90a6c13a97511fa332935002c066686348368d2"} Oct 30 01:27:57.232842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:57.232792 4905 generic.go:356] "Generic (PLEG): container finished" podID="7fc20abd-dca9-4273-bd5f-6d89ba9032d8" containerID="383406d1ae8c80d61f5d52d5dc5a1a1ae425568e35d12a56216d166a2fd47077" exitCode=0 Oct 30 01:27:57.233302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:57.232855 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" event={"ID":"7fc20abd-dca9-4273-bd5f-6d89ba9032d8","Type":"ContainerDied","Data":"383406d1ae8c80d61f5d52d5dc5a1a1ae425568e35d12a56216d166a2fd47077"} Oct 30 01:27:58.389294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.389271 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:58.459883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.459855 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-kube-api-access-vrsts\" (UniqueName: \"kubernetes.io/projected/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-kube-api-access-vrsts\") pod \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " Oct 30 01:27:58.460065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.459935 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-openshift-ca\") pod \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " Oct 30 01:27:58.460065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.460016 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-xks-ca\") pod \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\" (UID: \"7fc20abd-dca9-4273-bd5f-6d89ba9032d8\") " Oct 30 01:27:58.460380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.460341 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-openshift-ca" pod "7fc20abd-dca9-4273-bd5f-6d89ba9032d8" (UID: "7fc20abd-dca9-4273-bd5f-6d89ba9032d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:58.462243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.462206 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-xks-ca" pod "7fc20abd-dca9-4273-bd5f-6d89ba9032d8" (UID: "7fc20abd-dca9-4273-bd5f-6d89ba9032d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:58.462375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.462328 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-kube-api-access-vrsts" pod "7fc20abd-dca9-4273-bd5f-6d89ba9032d8" (UID: "7fc20abd-dca9-4273-bd5f-6d89ba9032d8"). InnerVolumeSpecName "kube-api-access-vrsts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:58.560987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.560900 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrsts\" (UniqueName: \"kubernetes.io/projected/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-kube-api-access-vrsts\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.560987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.560930 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.560987 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.560944 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc20abd-dca9-4273-bd5f-6d89ba9032d8-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.709380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.709345 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" podUID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:58.709380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:58.709363 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" podUID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:59.246811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:59.246776 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" Oct 30 01:27:59.247013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:59.246818 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x" event={"ID":"7fc20abd-dca9-4273-bd5f-6d89ba9032d8","Type":"ContainerDied","Data":"cee8ae27aa4e40c5057aeeffc90a6c13a97511fa332935002c066686348368d2"} Oct 30 01:27:59.247013 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:27:59.246911 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cee8ae27aa4e40c5057aeeffc90a6c13a97511fa332935002c066686348368d2" Oct 30 01:28:00.086077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.086046 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:28:00.132069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.132044 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:28:00.135213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.135192 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:28:00.179822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.179787 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-socket\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.179835 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istiod-ca-cert\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.179868 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-credential-socket\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.179921 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-token\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180020 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.179979 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-envoy\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180056 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-data\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180089 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-socket" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.180298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180099 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-kube-api-access-5sf42\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-kube-api-access-5sf42\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180156 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-podinfo\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180195 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-certs\") pod \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\" (UID: \"b0c229f1-7e63-41f2-977a-043fa4ca9c26\") " Oct 30 01:28:00.180298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180219 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-credential-socket" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.180707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180323 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-data" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.180707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180489 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-certs" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.180707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180606 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.180707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180624 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.180707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180633 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.180707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180642 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.180707 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.180677 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istiod-ca-cert" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:00.182607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.182580 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-token" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.182864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.182834 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-kube-api-access-5sf42" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "kube-api-access-5sf42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.183217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.183180 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-envoy" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.183217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.183190 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-podinfo" pod "b0c229f1-7e63-41f2-977a-043fa4ca9c26" (UID: "b0c229f1-7e63-41f2-977a-043fa4ca9c26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:00.256658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.256627 4905 generic.go:356] "Generic (PLEG): container finished" podID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerID="1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f" exitCode=0 Oct 30 01:28:00.256869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.256700 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" Oct 30 01:28:00.256869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.256728 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" event={"ID":"b0c229f1-7e63-41f2-977a-043fa4ca9c26","Type":"ContainerDied","Data":"1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f"} Oct 30 01:28:00.256869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.256800 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq" event={"ID":"b0c229f1-7e63-41f2-977a-043fa4ca9c26","Type":"ContainerDied","Data":"953666ba881ffe78b9db99e50c49942a4516ab797ed78a0a40ccfdaaabf9ef17"} Oct 30 01:28:00.256869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.256835 4905 scope.go:122] "RemoveContainer" containerID="1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f" Oct 30 01:28:00.258492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.258300 4905 generic.go:356] "Generic (PLEG): container finished" podID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerID="ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246" exitCode=0 Oct 30 01:28:00.258492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.258329 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" event={"ID":"e3ade1c6-0236-43a4-8e65-3644cb096c5c","Type":"ContainerDied","Data":"ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246"} Oct 30 01:28:00.258492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.258351 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" Oct 30 01:28:00.258492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.258386 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6" event={"ID":"e3ade1c6-0236-43a4-8e65-3644cb096c5c","Type":"ContainerDied","Data":"614e4012cd032b5d19f63f5255d46b6b0e61edfd2a51561feea67b09759488a6"} Oct 30 01:28:00.260197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.260082 4905 generic.go:356] "Generic (PLEG): container finished" podID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerID="1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8" exitCode=0 Oct 30 01:28:00.260197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.260165 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" Oct 30 01:28:00.260401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.260209 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" event={"ID":"3f736e26-d57f-48d1-992a-971e4d67c65e","Type":"ContainerDied","Data":"1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8"} Oct 30 01:28:00.260401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.260256 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27" event={"ID":"3f736e26-d57f-48d1-992a-971e4d67c65e","Type":"ContainerDied","Data":"a15fb9a2e23b15dbb7df19391aaac214f803d0c12eb23d5432463ce66367c72d"} Oct 30 01:28:00.268443 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.268426 4905 scope.go:122] "RemoveContainer" containerID="d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0" Oct 30 01:28:00.277311 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.277295 4905 scope.go:122] "RemoveContainer" containerID="1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f" Oct 30 01:28:00.277620 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:00.277601 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f\": container with ID starting with 1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f not found: ID does not exist" containerID="1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f" Oct 30 01:28:00.277683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.277630 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f"} err="failed to get container status \"1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f\": rpc error: code = NotFound desc = could not find container \"1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f\": container with ID starting with 1ae9afdb50e354dbddfab5350b54a3457bda2a838074e3c74369531f80bcdf1f not found: ID does not exist" Oct 30 01:28:00.277683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.277647 4905 scope.go:122] "RemoveContainer" containerID="d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0" Oct 30 01:28:00.277901 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:00.277872 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0\": container with ID starting with d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0 not found: ID does not exist" containerID="d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0" Oct 30 01:28:00.277976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.277911 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0"} err="failed to get container status \"d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0\": rpc error: code = NotFound desc = could not find container \"d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0\": container with ID starting with d0148f5a7bf8a1ceb04e7f5e3a3093613a93678b1a6e1652fd267bf02ca137c0 not found: ID does not exist" Oct 30 01:28:00.277976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.277932 4905 scope.go:122] "RemoveContainer" containerID="ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246" Oct 30 01:28:00.281125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281105 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-podinfo\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.281219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281145 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-credential-socket\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.281219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281190 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-token\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-token\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.281345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281221 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-certs\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.281345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281248 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-data\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.281345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281283 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f736e26-d57f-48d1-992a-971e4d67c65e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f736e26-d57f-48d1-992a-971e4d67c65e-istiod-ca-cert\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.281345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281324 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-data\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281360 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-socket\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281400 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-envoy\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281428 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-credential-socket\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281471 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-token\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281508 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-socket\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281534 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-credential-socket" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281548 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istiod-ca-cert\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281579 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-certs\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.281596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281597 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-certs" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281629 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-data" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281637 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-kube-api-access-gqljg\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-kube-api-access-gqljg\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281717 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-kube-api-access-x4jcx\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-kube-api-access-x4jcx\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281728 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-credential-socket" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281762 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-podinfo\") pod \"3f736e26-d57f-48d1-992a-971e4d67c65e\" (UID: \"3f736e26-d57f-48d1-992a-971e4d67c65e\") " Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281801 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-envoy\") pod \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\" (UID: \"e3ade1c6-0236-43a4-8e65-3644cb096c5c\") " Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.281949 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-socket" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282007 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-data" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.282263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282260 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282284 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282301 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282314 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282327 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282340 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282353 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282367 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282380 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c229f1-7e63-41f2-977a-043fa4ca9c26-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282394 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282419 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sf42\" (UniqueName: \"kubernetes.io/projected/b0c229f1-7e63-41f2-977a-043fa4ca9c26-kube-api-access-5sf42\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.282820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282664 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f736e26-d57f-48d1-992a-971e4d67c65e-istiod-ca-cert" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:00.283413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.282875 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-socket" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.283807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.283782 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-certs" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.283995 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.283975 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istiod-ca-cert" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:00.284242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.284219 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-envoy" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.284329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.284297 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-kube-api-access-gqljg" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "kube-api-access-gqljg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.285406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.285374 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-token" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.285495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.285421 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-kube-api-access-x4jcx" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "kube-api-access-x4jcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.285495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.285466 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-podinfo" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:00.285619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.285495 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-token" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.285988 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.285971 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-envoy" pod "e3ade1c6-0236-43a4-8e65-3644cb096c5c" (UID: "e3ade1c6-0236-43a4-8e65-3644cb096c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.286752 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.286730 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-podinfo" pod "3f736e26-d57f-48d1-992a-971e4d67c65e" (UID: "3f736e26-d57f-48d1-992a-971e4d67c65e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:00.289909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.289886 4905 scope.go:122] "RemoveContainer" containerID="ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246" Oct 30 01:28:00.290126 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:00.290105 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246\": container with ID starting with ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246 not found: ID does not exist" containerID="ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246" Oct 30 01:28:00.290208 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.290137 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246"} err="failed to get container status \"ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246\": rpc error: code = NotFound desc = could not find container \"ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246\": container with ID starting with ba9d23e4242c83f256de21b719c2307921a5276b4973adae12aef54138163246 not found: ID does not exist" Oct 30 01:28:00.290208 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.290159 4905 scope.go:122] "RemoveContainer" containerID="1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8" Oct 30 01:28:00.298513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.298495 4905 scope.go:122] "RemoveContainer" containerID="1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8" Oct 30 01:28:00.298807 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:00.298788 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8\": container with ID starting with 1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8 not found: ID does not exist" containerID="1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8" Oct 30 01:28:00.298877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.298817 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8"} err="failed to get container status \"1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8\": rpc error: code = NotFound desc = could not find container \"1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8\": container with ID starting with 1194e691f70f673232a593d7682ecea18e372dd4b944e7182baa6f110967f8b8 not found: ID does not exist" Oct 30 01:28:00.344650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.344578 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq"] Oct 30 01:28:00.348093 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.348062 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-6kbpq"] Oct 30 01:28:00.383411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383383 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383405 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383415 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f736e26-d57f-48d1-992a-971e4d67c65e-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383424 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383432 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383440 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383449 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383456 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383464 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqljg\" (UniqueName: \"kubernetes.io/projected/e3ade1c6-0236-43a4-8e65-3644cb096c5c-kube-api-access-gqljg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383475 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jcx\" (UniqueName: \"kubernetes.io/projected/3f736e26-d57f-48d1-992a-971e4d67c65e-kube-api-access-x4jcx\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383483 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f736e26-d57f-48d1-992a-971e4d67c65e-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.383642 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.383492 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ade1c6-0236-43a4-8e65-3644cb096c5c-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.609394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.609322 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27"] Oct 30 01:28:00.612920 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.612886 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-n7c27"] Oct 30 01:28:00.649102 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.649072 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6"] Oct 30 01:28:00.654763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:00.654736 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-rcrj6"] Oct 30 01:28:01.189169 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:01.189134 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" path="/var/lib/kubelet/pods/3f736e26-d57f-48d1-992a-971e4d67c65e/volumes" Oct 30 01:28:01.189671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:01.189655 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" path="/var/lib/kubelet/pods/b0c229f1-7e63-41f2-977a-043fa4ca9c26/volumes" Oct 30 01:28:01.190117 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:01.190101 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" path="/var/lib/kubelet/pods/e3ade1c6-0236-43a4-8e65-3644cb096c5c/volumes" Oct 30 01:28:05.793799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:05.793762 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5"] Oct 30 01:28:05.826798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:05.826763 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7"] Oct 30 01:28:07.545631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:07.545583 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x"] Oct 30 01:28:07.559101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:07.559072 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-954bf6-gj84x"] Oct 30 01:28:08.311579 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.311484 4905 generic.go:356] "Generic (PLEG): container finished" podID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" containerID="a491d5fce012bb888087a39d0268a06647b0215677b33a696cf7396297882fba" exitCode=0 Oct 30 01:28:08.311712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.311573 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" event={"ID":"211cc2b7-9c6a-4213-b954-51bea62aa9b1","Type":"ContainerDied","Data":"a491d5fce012bb888087a39d0268a06647b0215677b33a696cf7396297882fba"} Oct 30 01:28:08.313640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.313615 4905 generic.go:356] "Generic (PLEG): container finished" podID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" containerID="859296d7de1cec95e06d6ffe2d9ae43bbf40e0f246f8b600ab302f8a4ea7a55f" exitCode=0 Oct 30 01:28:08.313776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.313677 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" event={"ID":"f928f4d2-b9fc-4ad7-954f-6cc37c140272","Type":"ContainerDied","Data":"859296d7de1cec95e06d6ffe2d9ae43bbf40e0f246f8b600ab302f8a4ea7a55f"} Oct 30 01:28:08.347846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.347783 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:28:08.364638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.364610 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:28:08.463061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.463032 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211cc2b7-9c6a-4213-b954-51bea62aa9b1-kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/211cc2b7-9c6a-4213-b954-51bea62aa9b1-kube-api-access-twfm2\") pod \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " Oct 30 01:28:08.463280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.463070 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211cc2b7-9c6a-4213-b954-51bea62aa9b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211cc2b7-9c6a-4213-b954-51bea62aa9b1-plugins-config-volume\") pod \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\" (UID: \"211cc2b7-9c6a-4213-b954-51bea62aa9b1\") " Oct 30 01:28:08.463280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.463149 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f928f4d2-b9fc-4ad7-954f-6cc37c140272-kube-api-access-89tls\" (UniqueName: \"kubernetes.io/projected/f928f4d2-b9fc-4ad7-954f-6cc37c140272-kube-api-access-89tls\") pod \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " Oct 30 01:28:08.463280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.463206 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f928f4d2-b9fc-4ad7-954f-6cc37c140272-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f928f4d2-b9fc-4ad7-954f-6cc37c140272-plugins-config-volume\") pod \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\" (UID: \"f928f4d2-b9fc-4ad7-954f-6cc37c140272\") " Oct 30 01:28:08.463577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.463545 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211cc2b7-9c6a-4213-b954-51bea62aa9b1-plugins-config-volume" pod "211cc2b7-9c6a-4213-b954-51bea62aa9b1" (UID: "211cc2b7-9c6a-4213-b954-51bea62aa9b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:08.463720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.463652 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f928f4d2-b9fc-4ad7-954f-6cc37c140272-plugins-config-volume" pod "f928f4d2-b9fc-4ad7-954f-6cc37c140272" (UID: "f928f4d2-b9fc-4ad7-954f-6cc37c140272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:08.465502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.465476 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f928f4d2-b9fc-4ad7-954f-6cc37c140272-kube-api-access-89tls" pod "f928f4d2-b9fc-4ad7-954f-6cc37c140272" (UID: "f928f4d2-b9fc-4ad7-954f-6cc37c140272"). InnerVolumeSpecName "kube-api-access-89tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:08.465789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.465772 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211cc2b7-9c6a-4213-b954-51bea62aa9b1-kube-api-access-twfm2" pod "211cc2b7-9c6a-4213-b954-51bea62aa9b1" (UID: "211cc2b7-9c6a-4213-b954-51bea62aa9b1"). InnerVolumeSpecName "kube-api-access-twfm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:08.564537 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.564427 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89tls\" (UniqueName: \"kubernetes.io/projected/f928f4d2-b9fc-4ad7-954f-6cc37c140272-kube-api-access-89tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:08.564537 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.564462 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f928f4d2-b9fc-4ad7-954f-6cc37c140272-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:08.564537 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.564480 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/211cc2b7-9c6a-4213-b954-51bea62aa9b1-kube-api-access-twfm2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:08.564537 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:08.564495 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211cc2b7-9c6a-4213-b954-51bea62aa9b1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:09.188898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.188862 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc20abd-dca9-4273-bd5f-6d89ba9032d8" path="/var/lib/kubelet/pods/7fc20abd-dca9-4273-bd5f-6d89ba9032d8/volumes" Oct 30 01:28:09.320551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.320445 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" Oct 30 01:28:09.320551 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.320469 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk" event={"ID":"211cc2b7-9c6a-4213-b954-51bea62aa9b1","Type":"ContainerDied","Data":"0566c59764855c8d34cd883c7e2301497d939f712d4a79e64595560bce3c5060"} Oct 30 01:28:09.320816 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.320561 4905 scope.go:122] "RemoveContainer" containerID="a491d5fce012bb888087a39d0268a06647b0215677b33a696cf7396297882fba" Oct 30 01:28:09.322092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.322012 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" event={"ID":"f928f4d2-b9fc-4ad7-954f-6cc37c140272","Type":"ContainerDied","Data":"c23d110b4729133e79e20e64c727f97d0e5bb5d1075e398b56d90e3906d1f929"} Oct 30 01:28:09.322092 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.322052 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk" Oct 30 01:28:09.332443 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.332418 4905 scope.go:122] "RemoveContainer" containerID="859296d7de1cec95e06d6ffe2d9ae43bbf40e0f246f8b600ab302f8a4ea7a55f" Oct 30 01:28:09.375190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.375160 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk"] Oct 30 01:28:09.388261 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.388231 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-60f38f-58c456889d-dbmkk"] Oct 30 01:28:09.424206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.424174 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk"] Oct 30 01:28:09.428273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:09.428236 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-60f38f-7689ff574c-tsvjk"] Oct 30 01:28:10.794444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:10.794384 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" podUID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" containerName="payload-processing" containerID="cri-o://056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0" gracePeriod=25 Oct 30 01:28:10.827367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:10.827328 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" podUID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" containerName="payload-pre-processing" containerID="cri-o://9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4" gracePeriod=25 Oct 30 01:28:11.191201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:11.191123 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" path="/var/lib/kubelet/pods/211cc2b7-9c6a-4213-b954-51bea62aa9b1/volumes" Oct 30 01:28:11.191545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:11.191512 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" path="/var/lib/kubelet/pods/f928f4d2-b9fc-4ad7-954f-6cc37c140272/volumes" Oct 30 01:28:12.418631 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:12.418598 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:28:14.265559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.262369 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4"] Oct 30 01:28:14.265559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.262645 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" containerID="cri-o://25cf9ef9302ee56a56113bf18fd11493e3f14700dff903816468ec5f26d4eb94" gracePeriod=30 Oct 30 01:28:14.919481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.919443 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl"] Oct 30 01:28:14.920040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920014 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7fc20abd-dca9-4273-bd5f-6d89ba9032d8" containerName="revoke-keys" Oct 30 01:28:14.920040 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920040 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc20abd-dca9-4273-bd5f-6d89ba9032d8" containerName="revoke-keys" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920074 4905 cpu_manager.go:414] "containerMap: removing container" podUID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" containerName="payload-pre-processing" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920080 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" containerName="payload-pre-processing" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920090 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920095 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920114 4905 cpu_manager.go:414] "containerMap: removing container" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920120 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151a86fd-2313-46ba-9470-7fc0c8f22d1b" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920128 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920133 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920141 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920146 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920156 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" containerName="payload-processing" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920161 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" containerName="payload-processing" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920174 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" Oct 30 01:28:14.920217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920183 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" Oct 30 01:28:14.920797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920254 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="211cc2b7-9c6a-4213-b954-51bea62aa9b1" containerName="payload-pre-processing" Oct 30 01:28:14.920797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920270 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" Oct 30 01:28:14.920797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920280 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ade1c6-0236-43a4-8e65-3644cb096c5c" containerName="istio-proxy" Oct 30 01:28:14.920797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920288 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f928f4d2-b9fc-4ad7-954f-6cc37c140272" containerName="payload-processing" Oct 30 01:28:14.920797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920296 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f736e26-d57f-48d1-992a-971e4d67c65e" containerName="istio-proxy" Oct 30 01:28:14.920797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.920304 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc20abd-dca9-4273-bd5f-6d89ba9032d8" containerName="revoke-keys" Oct 30 01:28:14.925269 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.925250 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:14.939856 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.939822 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" podUID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:28:14.942087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.942060 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:28:14.944066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.944036 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl"] Oct 30 01:28:14.949505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.949478 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" podUID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:28:14.955512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.955480 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:28:14.972809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.972547 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:28:14.972809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:14.972598 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:28:15.033760 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.033726 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.033909 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.033874 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf496\" (UniqueName: \"kubernetes.io/projected/7a448126-7102-44cb-9e1e-5deb989de28b-kube-api-access-xf496\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.033976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.033911 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.134999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.134957 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf496\" (UniqueName: \"kubernetes.io/projected/7a448126-7102-44cb-9e1e-5deb989de28b-kube-api-access-xf496\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.135215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.135023 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.135215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.135100 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.135857 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.135824 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.137821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.137793 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.155576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.155546 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf496\" (UniqueName: \"kubernetes.io/projected/7a448126-7102-44cb-9e1e-5deb989de28b-kube-api-access-xf496\") pod \"maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.241052 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.241017 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:15.543355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:15.543323 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl"] Oct 30 01:28:15.544166 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:28:15.544137 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a448126_7102_44cb_9e1e_5deb989de28b.slice/crio-af0c27db673dbffbbea6f4fe55ae3a6f6e8a3dcaf419d124bd3c7eb4eae9d5b0 WatchSource:0}: Error finding container af0c27db673dbffbbea6f4fe55ae3a6f6e8a3dcaf419d124bd3c7eb4eae9d5b0: Status 404 returned error can't find the container with id af0c27db673dbffbbea6f4fe55ae3a6f6e8a3dcaf419d124bd3c7eb4eae9d5b0 Oct 30 01:28:16.362948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:16.362908 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" event={"ID":"7a448126-7102-44cb-9e1e-5deb989de28b","Type":"ContainerStarted","Data":"d32551c9b734dc710f92e1887971316a330b124ffeda442132653924adc5e090"} Oct 30 01:28:16.363120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:16.362967 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" event={"ID":"7a448126-7102-44cb-9e1e-5deb989de28b","Type":"ContainerStarted","Data":"af0c27db673dbffbbea6f4fe55ae3a6f6e8a3dcaf419d124bd3c7eb4eae9d5b0"} Oct 30 01:28:16.406895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:16.406841 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" podStartSLOduration=2.406826708 podStartE2EDuration="2.406826708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:16.405045487 +0000 UTC m=+2981.348218681" watchObservedRunningTime="2026-10-30 01:28:16.406826708 +0000 UTC m=+2981.349999902" Oct 30 01:28:17.373073 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:17.373037 4905 generic.go:356] "Generic (PLEG): container finished" podID="7a448126-7102-44cb-9e1e-5deb989de28b" containerID="d32551c9b734dc710f92e1887971316a330b124ffeda442132653924adc5e090" exitCode=0 Oct 30 01:28:17.373564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:17.373105 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" event={"ID":"7a448126-7102-44cb-9e1e-5deb989de28b","Type":"ContainerDied","Data":"d32551c9b734dc710f92e1887971316a330b124ffeda442132653924adc5e090"} Oct 30 01:28:18.524025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.523999 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:18.569713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.569677 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a448126-7102-44cb-9e1e-5deb989de28b-kube-api-access-xf496\" (UniqueName: \"kubernetes.io/projected/7a448126-7102-44cb-9e1e-5deb989de28b-kube-api-access-xf496\") pod \"7a448126-7102-44cb-9e1e-5deb989de28b\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " Oct 30 01:28:18.570096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.570074 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-xks-ca\") pod \"7a448126-7102-44cb-9e1e-5deb989de28b\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " Oct 30 01:28:18.570194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.570143 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-openshift-ca\") pod \"7a448126-7102-44cb-9e1e-5deb989de28b\" (UID: \"7a448126-7102-44cb-9e1e-5deb989de28b\") " Oct 30 01:28:18.570615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.570589 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-openshift-ca" pod "7a448126-7102-44cb-9e1e-5deb989de28b" (UID: "7a448126-7102-44cb-9e1e-5deb989de28b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:18.572475 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.572448 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a448126-7102-44cb-9e1e-5deb989de28b-kube-api-access-xf496" pod "7a448126-7102-44cb-9e1e-5deb989de28b" (UID: "7a448126-7102-44cb-9e1e-5deb989de28b"). InnerVolumeSpecName "kube-api-access-xf496". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:18.573930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.573904 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-xks-ca" pod "7a448126-7102-44cb-9e1e-5deb989de28b" (UID: "7a448126-7102-44cb-9e1e-5deb989de28b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:18.654690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.654613 4905 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:28:18.654690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.654668 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:28:18.671892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.671864 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:18.671892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.671888 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a448126-7102-44cb-9e1e-5deb989de28b-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:18.672071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:18.671900 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf496\" (UniqueName: \"kubernetes.io/projected/7a448126-7102-44cb-9e1e-5deb989de28b-kube-api-access-xf496\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.384361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.384320 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" Oct 30 01:28:19.384487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.384319 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl" event={"ID":"7a448126-7102-44cb-9e1e-5deb989de28b","Type":"ContainerDied","Data":"af0c27db673dbffbbea6f4fe55ae3a6f6e8a3dcaf419d124bd3c7eb4eae9d5b0"} Oct 30 01:28:19.384487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.384463 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0c27db673dbffbbea6f4fe55ae3a6f6e8a3dcaf419d124bd3c7eb4eae9d5b0" Oct 30 01:28:19.386335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.386310 4905 generic.go:356] "Generic (PLEG): container finished" podID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerID="25cf9ef9302ee56a56113bf18fd11493e3f14700dff903816468ec5f26d4eb94" exitCode=0 Oct 30 01:28:19.386571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.386391 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" event={"ID":"dde15e70-d861-4e0e-b36a-151e0ac677af","Type":"ContainerDied","Data":"25cf9ef9302ee56a56113bf18fd11493e3f14700dff903816468ec5f26d4eb94"} Oct 30 01:28:19.518001 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.517978 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:28:19.581353 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.581314 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-credential-socket\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.581871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.581404 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-kube-api-access-gkj7g\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-kube-api-access-gkj7g\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.581871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.581449 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde15e70-d861-4e0e-b36a-151e0ac677af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde15e70-d861-4e0e-b36a-151e0ac677af-istiod-ca-cert\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.581871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.581476 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-certs\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.582065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.581925 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-credential-socket" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:19.582065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582012 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-socket\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.582189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582077 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-podinfo\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.582189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582176 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-data\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.582299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582212 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-envoy\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.582299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582256 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-token\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-token\") pod \"dde15e70-d861-4e0e-b36a-151e0ac677af\" (UID: \"dde15e70-d861-4e0e-b36a-151e0ac677af\") " Oct 30 01:28:19.582299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582260 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde15e70-d861-4e0e-b36a-151e0ac677af-istiod-ca-cert" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:19.582820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582796 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.583012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582824 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde15e70-d861-4e0e-b36a-151e0ac677af-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.583012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582874 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-data" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:19.583012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.582910 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-socket" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:19.583249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.583155 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-certs" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:19.584605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.584571 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-kube-api-access-gkj7g" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "kube-api-access-gkj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:19.585209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.585178 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-envoy" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:19.585209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.585196 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-token" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:19.586813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.586789 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-podinfo" pod "dde15e70-d861-4e0e-b36a-151e0ac677af" (UID: "dde15e70-d861-4e0e-b36a-151e0ac677af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:19.683827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.683786 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkj7g\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-kube-api-access-gkj7g\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.683827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.683822 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.683827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.683832 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.684079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.683842 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.684079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.683851 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.684079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.683859 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.684079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.683867 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde15e70-d861-4e0e-b36a-151e0ac677af-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:19.728656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.728619 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2"] Oct 30 01:28:19.737455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:19.737419 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6"] Oct 30 01:28:20.393822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:20.393795 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" Oct 30 01:28:20.394023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:20.393795 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4" event={"ID":"dde15e70-d861-4e0e-b36a-151e0ac677af","Type":"ContainerDied","Data":"4ee3786762f890161df152af76707a4cf87c60c9535f11f65544404c96b3c609"} Oct 30 01:28:20.394023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:20.393943 4905 scope.go:122] "RemoveContainer" containerID="25cf9ef9302ee56a56113bf18fd11493e3f14700dff903816468ec5f26d4eb94" Oct 30 01:28:20.448488 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:20.448454 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4"] Oct 30 01:28:20.461804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:20.461773 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-n2bv4"] Oct 30 01:28:21.190125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:21.190084 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" path="/var/lib/kubelet/pods/dde15e70-d861-4e0e-b36a-151e0ac677af/volumes" Oct 30 01:28:24.939891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:24.939852 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" podUID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:28:24.948917 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:24.948887 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" podUID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:28:27.471552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.471502 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj"] Oct 30 01:28:27.472044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.471939 4905 cpu_manager.go:414] "containerMap: removing container" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" Oct 30 01:28:27.472044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.471949 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" Oct 30 01:28:27.472044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.471965 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7a448126-7102-44cb-9e1e-5deb989de28b" containerName="revoke-keys" Oct 30 01:28:27.472044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.471970 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a448126-7102-44cb-9e1e-5deb989de28b" containerName="revoke-keys" Oct 30 01:28:27.472044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.472038 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde15e70-d861-4e0e-b36a-151e0ac677af" containerName="istio-proxy" Oct 30 01:28:27.472044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.472045 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c229f1-7e63-41f2-977a-043fa4ca9c26" containerName="istio-proxy" Oct 30 01:28:27.472505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.472052 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a448126-7102-44cb-9e1e-5deb989de28b" containerName="revoke-keys" Oct 30 01:28:27.476547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.476508 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.494881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.494834 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj"] Oct 30 01:28:27.506081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.506052 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:28:27.506449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.506414 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:28:27.506602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.506486 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:28:27.506602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.506597 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:28:27.560193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.560161 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtx9\" (UniqueName: \"kubernetes.io/projected/5af47d61-5943-430a-9a1b-a8f7f12a952a-kube-api-access-kwtx9\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.560371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.560225 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.560424 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.560402 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.661462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.661418 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.661462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.661465 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtx9\" (UniqueName: \"kubernetes.io/projected/5af47d61-5943-430a-9a1b-a8f7f12a952a-kube-api-access-kwtx9\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.661754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.661546 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.662227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.662203 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.663965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.663939 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.681965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.681935 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtx9\" (UniqueName: \"kubernetes.io/projected/5af47d61-5943-430a-9a1b-a8f7f12a952a-kube-api-access-kwtx9\") pod \"maas-api-revoke-sub-127241af-scgpj\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:27.789703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:27.789669 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:28.082694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:28.082663 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj"] Oct 30 01:28:28.083644 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:28:28.083620 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af47d61_5943_430a_9a1b_a8f7f12a952a.slice/crio-f962d20179d1404a00afdbe8c48e7317c6254d2f2e161a479bff03b00ba94610 WatchSource:0}: Error finding container f962d20179d1404a00afdbe8c48e7317c6254d2f2e161a479bff03b00ba94610: Status 404 returned error can't find the container with id f962d20179d1404a00afdbe8c48e7317c6254d2f2e161a479bff03b00ba94610 Oct 30 01:28:28.438438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:28.438392 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" event={"ID":"5af47d61-5943-430a-9a1b-a8f7f12a952a","Type":"ContainerStarted","Data":"69741ac64a4f5321d9da12ed5635c20132364cc40b8a1dc94a3ebf72ee1f2fcd"} Oct 30 01:28:28.438662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:28.438459 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" event={"ID":"5af47d61-5943-430a-9a1b-a8f7f12a952a","Type":"ContainerStarted","Data":"f962d20179d1404a00afdbe8c48e7317c6254d2f2e161a479bff03b00ba94610"} Oct 30 01:28:28.478406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:28.478349 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" podStartSLOduration=1.478334295 podStartE2EDuration="1.478334295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:28.477713252 +0000 UTC m=+2993.420886444" watchObservedRunningTime="2026-10-30 01:28:28.478334295 +0000 UTC m=+2993.421507542" Oct 30 01:28:29.444677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:29.444642 4905 generic.go:356] "Generic (PLEG): container finished" podID="5af47d61-5943-430a-9a1b-a8f7f12a952a" containerID="69741ac64a4f5321d9da12ed5635c20132364cc40b8a1dc94a3ebf72ee1f2fcd" exitCode=0 Oct 30 01:28:29.444954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:29.444696 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" event={"ID":"5af47d61-5943-430a-9a1b-a8f7f12a952a","Type":"ContainerDied","Data":"69741ac64a4f5321d9da12ed5635c20132364cc40b8a1dc94a3ebf72ee1f2fcd"} Oct 30 01:28:30.584333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.584308 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:30.689630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.689594 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af47d61-5943-430a-9a1b-a8f7f12a952a-kube-api-access-kwtx9\" (UniqueName: \"kubernetes.io/projected/5af47d61-5943-430a-9a1b-a8f7f12a952a-kube-api-access-kwtx9\") pod \"5af47d61-5943-430a-9a1b-a8f7f12a952a\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " Oct 30 01:28:30.689822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.689754 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-openshift-ca\") pod \"5af47d61-5943-430a-9a1b-a8f7f12a952a\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " Oct 30 01:28:30.689890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.689836 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-xks-ca\") pod \"5af47d61-5943-430a-9a1b-a8f7f12a952a\" (UID: \"5af47d61-5943-430a-9a1b-a8f7f12a952a\") " Oct 30 01:28:30.690130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.690104 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-openshift-ca" pod "5af47d61-5943-430a-9a1b-a8f7f12a952a" (UID: "5af47d61-5943-430a-9a1b-a8f7f12a952a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:30.691975 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.691952 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-xks-ca" pod "5af47d61-5943-430a-9a1b-a8f7f12a952a" (UID: "5af47d61-5943-430a-9a1b-a8f7f12a952a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:30.692052 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.691971 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af47d61-5943-430a-9a1b-a8f7f12a952a-kube-api-access-kwtx9" pod "5af47d61-5943-430a-9a1b-a8f7f12a952a" (UID: "5af47d61-5943-430a-9a1b-a8f7f12a952a"). InnerVolumeSpecName "kube-api-access-kwtx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:30.790808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.790767 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtx9\" (UniqueName: \"kubernetes.io/projected/5af47d61-5943-430a-9a1b-a8f7f12a952a-kube-api-access-kwtx9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:30.790808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.790804 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:30.791023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:30.790819 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af47d61-5943-430a-9a1b-a8f7f12a952a-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:31.060258 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.060227 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:28:31.117572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.114267 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:28:31.194867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.194839 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff91ee3-305c-4562-a05b-4eac7828ef1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff91ee3-305c-4562-a05b-4eac7828ef1c-plugins-config-volume\") pod \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " Oct 30 01:28:31.195062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.194880 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff91ee3-305c-4562-a05b-4eac7828ef1c-kube-api-access-z2h8t\" (UniqueName: \"kubernetes.io/projected/1ff91ee3-305c-4562-a05b-4eac7828ef1c-kube-api-access-z2h8t\") pod \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\" (UID: \"1ff91ee3-305c-4562-a05b-4eac7828ef1c\") " Oct 30 01:28:31.195062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.194927 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-kube-api-access-qv9g6\" (UniqueName: \"kubernetes.io/projected/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-kube-api-access-qv9g6\") pod \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " Oct 30 01:28:31.195062 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.195012 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-plugins-config-volume\") pod \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\" (UID: \"6fd0d9fb-f75a-4f0a-b984-1f64ad207014\") " Oct 30 01:28:31.195324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.195296 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff91ee3-305c-4562-a05b-4eac7828ef1c-plugins-config-volume" pod "1ff91ee3-305c-4562-a05b-4eac7828ef1c" (UID: "1ff91ee3-305c-4562-a05b-4eac7828ef1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:31.195407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.195365 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-plugins-config-volume" pod "6fd0d9fb-f75a-4f0a-b984-1f64ad207014" (UID: "6fd0d9fb-f75a-4f0a-b984-1f64ad207014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:31.198194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.198171 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff91ee3-305c-4562-a05b-4eac7828ef1c-kube-api-access-z2h8t" pod "1ff91ee3-305c-4562-a05b-4eac7828ef1c" (UID: "1ff91ee3-305c-4562-a05b-4eac7828ef1c"). InnerVolumeSpecName "kube-api-access-z2h8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:31.198447 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.198410 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-kube-api-access-qv9g6" pod "6fd0d9fb-f75a-4f0a-b984-1f64ad207014" (UID: "6fd0d9fb-f75a-4f0a-b984-1f64ad207014"). InnerVolumeSpecName "kube-api-access-qv9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:31.295895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.295862 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2h8t\" (UniqueName: \"kubernetes.io/projected/1ff91ee3-305c-4562-a05b-4eac7828ef1c-kube-api-access-z2h8t\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:31.296087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.296021 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9g6\" (UniqueName: \"kubernetes.io/projected/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-kube-api-access-qv9g6\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:31.296087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.296042 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd0d9fb-f75a-4f0a-b984-1f64ad207014-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:31.296087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.296056 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff91ee3-305c-4562-a05b-4eac7828ef1c-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:31.457083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.456996 4905 generic.go:356] "Generic (PLEG): container finished" podID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" containerID="056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0" exitCode=0 Oct 30 01:28:31.457083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.457054 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" Oct 30 01:28:31.457307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.457086 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" event={"ID":"1ff91ee3-305c-4562-a05b-4eac7828ef1c","Type":"ContainerDied","Data":"056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0"} Oct 30 01:28:31.457307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.457165 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5" event={"ID":"1ff91ee3-305c-4562-a05b-4eac7828ef1c","Type":"ContainerDied","Data":"9fc541c03ac576eccb21094e38c5109a10ddda5e38a2290a83d8998639e49eb0"} Oct 30 01:28:31.457307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.457200 4905 scope.go:122] "RemoveContainer" containerID="056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0" Oct 30 01:28:31.458711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.458558 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" Oct 30 01:28:31.458711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.458553 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj" event={"ID":"5af47d61-5943-430a-9a1b-a8f7f12a952a","Type":"ContainerDied","Data":"f962d20179d1404a00afdbe8c48e7317c6254d2f2e161a479bff03b00ba94610"} Oct 30 01:28:31.458826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.458731 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f962d20179d1404a00afdbe8c48e7317c6254d2f2e161a479bff03b00ba94610" Oct 30 01:28:31.460087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.460064 4905 generic.go:356] "Generic (PLEG): container finished" podID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" containerID="9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4" exitCode=0 Oct 30 01:28:31.460221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.460112 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" event={"ID":"6fd0d9fb-f75a-4f0a-b984-1f64ad207014","Type":"ContainerDied","Data":"9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4"} Oct 30 01:28:31.460221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.460133 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" Oct 30 01:28:31.460221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.460177 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7" event={"ID":"6fd0d9fb-f75a-4f0a-b984-1f64ad207014","Type":"ContainerDied","Data":"869c4d131830f36be01bbd7f22f2d34d0d6d016855f46f97d1a982a8ef1df0dc"} Oct 30 01:28:31.467459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.467437 4905 scope.go:122] "RemoveContainer" containerID="056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0" Oct 30 01:28:31.467757 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:31.467733 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0\": container with ID starting with 056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0 not found: ID does not exist" containerID="056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0" Oct 30 01:28:31.467837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.467764 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0"} err="failed to get container status \"056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0\": rpc error: code = NotFound desc = could not find container \"056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0\": container with ID starting with 056b58d06522878dfb785c15a485fd71fa8d9e5de8444a56461625032e619ea0 not found: ID does not exist" Oct 30 01:28:31.467837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.467787 4905 scope.go:122] "RemoveContainer" containerID="9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4" Oct 30 01:28:31.477074 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.477052 4905 scope.go:122] "RemoveContainer" containerID="9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4" Oct 30 01:28:31.477336 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:31.477318 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4\": container with ID starting with 9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4 not found: ID does not exist" containerID="9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4" Oct 30 01:28:31.477379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.477344 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4"} err="failed to get container status \"9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4\": rpc error: code = NotFound desc = could not find container \"9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4\": container with ID starting with 9ac75588799eb1e5e44a13c80ed8968300f3c6bf4af57011926b19ad2cf10bb4 not found: ID does not exist" Oct 30 01:28:31.514430 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.514397 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7"] Oct 30 01:28:31.520015 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.519996 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-954bf6-5fd9b95dcf-7pvk7"] Oct 30 01:28:31.556227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.556203 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5"] Oct 30 01:28:31.559608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:31.559585 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-954bf6-78757fc698-2tqd5"] Oct 30 01:28:33.189959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:33.189928 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" path="/var/lib/kubelet/pods/1ff91ee3-305c-4562-a05b-4eac7828ef1c/volumes" Oct 30 01:28:33.190360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:33.190294 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" path="/var/lib/kubelet/pods/6fd0d9fb-f75a-4f0a-b984-1f64ad207014/volumes" Oct 30 01:28:34.751484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:34.751434 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" containerID="cri-o://a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96" gracePeriod=45 Oct 30 01:28:34.757444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:34.757402 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" containerID="cri-o://12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e" gracePeriod=45 Oct 30 01:28:34.899853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:34.899813 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": dial tcp 10.132.0.90:8000: connect: connection refused" Oct 30 01:28:34.952196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:34.952157 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": dial tcp 10.132.0.69:8000: connect: connection refused" Oct 30 01:28:35.011900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.011841 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:28:35.046243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.046207 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:28:35.129241 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129211 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab69f74-0877-4f68-a0dd-b4a631fead70-kube-api-access-4bh2z\" (UniqueName: \"kubernetes.io/projected/3ab69f74-0877-4f68-a0dd-b4a631fead70-kube-api-access-4bh2z\") pod \"3ab69f74-0877-4f68-a0dd-b4a631fead70\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " Oct 30 01:28:35.129427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129251 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab69f74-0877-4f68-a0dd-b4a631fead70-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab69f74-0877-4f68-a0dd-b4a631fead70-tls-certs\") pod \"3ab69f74-0877-4f68-a0dd-b4a631fead70\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " Oct 30 01:28:35.129427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129314 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-model-cache\") pod \"3ab69f74-0877-4f68-a0dd-b4a631fead70\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " Oct 30 01:28:35.129427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129332 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d3906c-bd1a-4817-afc3-bb8158310757-kube-api-access-pmwzn\" (UniqueName: \"kubernetes.io/projected/b2d3906c-bd1a-4817-afc3-bb8158310757-kube-api-access-pmwzn\") pod \"b2d3906c-bd1a-4817-afc3-bb8158310757\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " Oct 30 01:28:35.129427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129366 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d3906c-bd1a-4817-afc3-bb8158310757-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d3906c-bd1a-4817-afc3-bb8158310757-tls-certs\") pod \"b2d3906c-bd1a-4817-afc3-bb8158310757\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " Oct 30 01:28:35.129427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129390 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-home\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-home\") pod \"b2d3906c-bd1a-4817-afc3-bb8158310757\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " Oct 30 01:28:35.129743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129443 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-tmp-dir\") pod \"b2d3906c-bd1a-4817-afc3-bb8158310757\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " Oct 30 01:28:35.129743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129479 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-model-cache\") pod \"b2d3906c-bd1a-4817-afc3-bb8158310757\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " Oct 30 01:28:35.129743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129559 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-dshm\") pod \"3ab69f74-0877-4f68-a0dd-b4a631fead70\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " Oct 30 01:28:35.129743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129590 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-dshm\") pod \"b2d3906c-bd1a-4817-afc3-bb8158310757\" (UID: \"b2d3906c-bd1a-4817-afc3-bb8158310757\") " Oct 30 01:28:35.129743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129625 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-home\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-home\") pod \"3ab69f74-0877-4f68-a0dd-b4a631fead70\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " Oct 30 01:28:35.129743 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129661 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-tmp-dir\") pod \"3ab69f74-0877-4f68-a0dd-b4a631fead70\" (UID: \"3ab69f74-0877-4f68-a0dd-b4a631fead70\") " Oct 30 01:28:35.130165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.129992 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-model-cache" pod "3ab69f74-0877-4f68-a0dd-b4a631fead70" (UID: "3ab69f74-0877-4f68-a0dd-b4a631fead70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.130165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.130152 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-tmp-dir" pod "b2d3906c-bd1a-4817-afc3-bb8158310757" (UID: "b2d3906c-bd1a-4817-afc3-bb8158310757"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.130294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.130182 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-home" pod "b2d3906c-bd1a-4817-afc3-bb8158310757" (UID: "b2d3906c-bd1a-4817-afc3-bb8158310757"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.130294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.130239 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-model-cache" pod "b2d3906c-bd1a-4817-afc3-bb8158310757" (UID: "b2d3906c-bd1a-4817-afc3-bb8158310757"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.130438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.130383 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-tmp-dir" pod "3ab69f74-0877-4f68-a0dd-b4a631fead70" (UID: "3ab69f74-0877-4f68-a0dd-b4a631fead70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.130502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.130473 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-home" pod "3ab69f74-0877-4f68-a0dd-b4a631fead70" (UID: "3ab69f74-0877-4f68-a0dd-b4a631fead70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.132364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.132329 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab69f74-0877-4f68-a0dd-b4a631fead70-tls-certs" pod "3ab69f74-0877-4f68-a0dd-b4a631fead70" (UID: "3ab69f74-0877-4f68-a0dd-b4a631fead70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:35.132499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.132429 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d3906c-bd1a-4817-afc3-bb8158310757-tls-certs" pod "b2d3906c-bd1a-4817-afc3-bb8158310757" (UID: "b2d3906c-bd1a-4817-afc3-bb8158310757"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:35.132703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.132671 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-dshm" pod "b2d3906c-bd1a-4817-afc3-bb8158310757" (UID: "b2d3906c-bd1a-4817-afc3-bb8158310757"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.132906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.132882 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-dshm" pod "3ab69f74-0877-4f68-a0dd-b4a631fead70" (UID: "3ab69f74-0877-4f68-a0dd-b4a631fead70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:35.133194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.133165 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab69f74-0877-4f68-a0dd-b4a631fead70-kube-api-access-4bh2z" pod "3ab69f74-0877-4f68-a0dd-b4a631fead70" (UID: "3ab69f74-0877-4f68-a0dd-b4a631fead70"). InnerVolumeSpecName "kube-api-access-4bh2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:35.133314 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.133298 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d3906c-bd1a-4817-afc3-bb8158310757-kube-api-access-pmwzn" pod "b2d3906c-bd1a-4817-afc3-bb8158310757" (UID: "b2d3906c-bd1a-4817-afc3-bb8158310757"). InnerVolumeSpecName "kube-api-access-pmwzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:35.230952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230918 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.230952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230946 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bh2z\" (UniqueName: \"kubernetes.io/projected/3ab69f74-0877-4f68-a0dd-b4a631fead70-kube-api-access-4bh2z\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.230952 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230955 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab69f74-0877-4f68-a0dd-b4a631fead70-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230963 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230973 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmwzn\" (UniqueName: \"kubernetes.io/projected/b2d3906c-bd1a-4817-afc3-bb8158310757-kube-api-access-pmwzn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230981 4905 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d3906c-bd1a-4817-afc3-bb8158310757-tls-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230988 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.230997 4905 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-tmp-dir\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.231004 4905 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-model-cache\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.231011 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.231018 4905 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d3906c-bd1a-4817-afc3-bb8158310757-dshm\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.231170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.231025 4905 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab69f74-0877-4f68-a0dd-b4a631fead70-home\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:35.484299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.484265 4905 generic.go:356] "Generic (PLEG): container finished" podID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerID="a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96" exitCode=0 Oct 30 01:28:35.484547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.484326 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" Oct 30 01:28:35.484547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.484367 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" event={"ID":"3ab69f74-0877-4f68-a0dd-b4a631fead70","Type":"ContainerDied","Data":"a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96"} Oct 30 01:28:35.484547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.484421 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2" event={"ID":"3ab69f74-0877-4f68-a0dd-b4a631fead70","Type":"ContainerDied","Data":"e4ea840dbaad991eabf1d87eac03cbe573dd1b9bf5f91d2e849413a208951f18"} Oct 30 01:28:35.484547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.484445 4905 scope.go:122] "RemoveContainer" containerID="a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96" Oct 30 01:28:35.485799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.485770 4905 generic.go:356] "Generic (PLEG): container finished" podID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerID="12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e" exitCode=0 Oct 30 01:28:35.485931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.485833 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" Oct 30 01:28:35.485931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.485876 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" event={"ID":"b2d3906c-bd1a-4817-afc3-bb8158310757","Type":"ContainerDied","Data":"12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e"} Oct 30 01:28:35.485931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.485910 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6" event={"ID":"b2d3906c-bd1a-4817-afc3-bb8158310757","Type":"ContainerDied","Data":"13b496ae0b957df6a9b1d3673660b8f3cca1613f9b4a025edb068f0728da7d4d"} Oct 30 01:28:35.493894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.493876 4905 scope.go:122] "RemoveContainer" containerID="a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96" Oct 30 01:28:35.494174 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:35.494149 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96\": container with ID starting with a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96 not found: ID does not exist" containerID="a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96" Oct 30 01:28:35.494219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.494183 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96"} err="failed to get container status \"a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96\": rpc error: code = NotFound desc = could not find container \"a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96\": container with ID starting with a397e1bf6109cd618a16598f7463e2bf1bea0a0a0f661b960cf1e55b0bf1ce96 not found: ID does not exist" Oct 30 01:28:35.494219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.494200 4905 scope.go:122] "RemoveContainer" containerID="12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e" Oct 30 01:28:35.502043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.502017 4905 scope.go:122] "RemoveContainer" containerID="12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e" Oct 30 01:28:35.502319 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:35.502293 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e\": container with ID starting with 12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e not found: ID does not exist" containerID="12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e" Oct 30 01:28:35.502367 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.502327 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e"} err="failed to get container status \"12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e\": rpc error: code = NotFound desc = could not find container \"12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e\": container with ID starting with 12151d5e9f10a6fd8500f395531c10a48131868c1b7ad1d6e11e2b3a8ed8ff5e not found: ID does not exist" Oct 30 01:28:35.539206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.539180 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2"] Oct 30 01:28:35.542465 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.542440 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/distinct-w4-b565ec-kserve-6bc489b89-sj4l2"] Oct 30 01:28:35.579096 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.579074 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6"] Oct 30 01:28:35.580402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:35.580381 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/premium-w4-b565ec-kserve-d4664b57d-q9vw6"] Oct 30 01:28:36.509490 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:36.509463 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:28:36.510861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:36.510838 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:28:37.189810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:37.189777 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" path="/var/lib/kubelet/pods/3ab69f74-0877-4f68-a0dd-b4a631fead70/volumes" Oct 30 01:28:37.190186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:37.190171 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" path="/var/lib/kubelet/pods/b2d3906c-bd1a-4817-afc3-bb8158310757/volumes" Oct 30 01:28:42.413370 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:42.413326 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:28:43.177598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:43.177563 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl"] Oct 30 01:28:43.191149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:43.191117 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b565ec-nn2nl"] Oct 30 01:28:45.188647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:45.188569 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a448126-7102-44cb-9e1e-5deb989de28b" path="/var/lib/kubelet/pods/7a448126-7102-44cb-9e1e-5deb989de28b/volumes" Oct 30 01:28:52.149954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.149917 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-jn4xq"] Oct 30 01:28:52.150495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.150182 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" podUID="5b38f224-a70d-4605-a35b-c4ae45ff122a" containerName="manager" containerID="cri-o://938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03" gracePeriod=30 Oct 30 01:28:52.412544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.412462 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 01:28:52.491570 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.491509 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-metrics-certs\") pod \"5b38f224-a70d-4605-a35b-c4ae45ff122a\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " Oct 30 01:28:52.491749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.491580 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-webhook-cert\") pod \"5b38f224-a70d-4605-a35b-c4ae45ff122a\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " Oct 30 01:28:52.491749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.491683 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b38f224-a70d-4605-a35b-c4ae45ff122a-kube-api-access-tzr2q\" (UniqueName: \"kubernetes.io/projected/5b38f224-a70d-4605-a35b-c4ae45ff122a-kube-api-access-tzr2q\") pod \"5b38f224-a70d-4605-a35b-c4ae45ff122a\" (UID: \"5b38f224-a70d-4605-a35b-c4ae45ff122a\") " Oct 30 01:28:52.493925 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.493889 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-metrics-certs" pod "5b38f224-a70d-4605-a35b-c4ae45ff122a" (UID: "5b38f224-a70d-4605-a35b-c4ae45ff122a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:52.494278 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.494242 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-webhook-cert" pod "5b38f224-a70d-4605-a35b-c4ae45ff122a" (UID: "5b38f224-a70d-4605-a35b-c4ae45ff122a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:52.494404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.494307 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b38f224-a70d-4605-a35b-c4ae45ff122a-kube-api-access-tzr2q" pod "5b38f224-a70d-4605-a35b-c4ae45ff122a" (UID: "5b38f224-a70d-4605-a35b-c4ae45ff122a"). InnerVolumeSpecName "kube-api-access-tzr2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:52.577936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.577903 4905 generic.go:356] "Generic (PLEG): container finished" podID="5b38f224-a70d-4605-a35b-c4ae45ff122a" containerID="938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03" exitCode=0 Oct 30 01:28:52.578127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.577954 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" Oct 30 01:28:52.578127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.577983 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" event={"ID":"5b38f224-a70d-4605-a35b-c4ae45ff122a","Type":"ContainerDied","Data":"938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03"} Oct 30 01:28:52.578127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.578039 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-jn4xq" event={"ID":"5b38f224-a70d-4605-a35b-c4ae45ff122a","Type":"ContainerDied","Data":"8f74f0009132fe6cc68473adc675bd04df1bc8c51850586a0ae519ee143e0285"} Oct 30 01:28:52.578127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.578075 4905 scope.go:122] "RemoveContainer" containerID="938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03" Oct 30 01:28:52.588963 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.588945 4905 scope.go:122] "RemoveContainer" containerID="938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03" Oct 30 01:28:52.589221 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:28:52.589198 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03\": container with ID starting with 938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03 not found: ID does not exist" containerID="938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03" Oct 30 01:28:52.589283 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.589232 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03"} err="failed to get container status \"938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03\": rpc error: code = NotFound desc = could not find container \"938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03\": container with ID starting with 938f1eda1c225e501a3191b3666a8c28a924ff6602e62622ed572c8cda66ca03 not found: ID does not exist" Oct 30 01:28:52.592687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.592668 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:52.592749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.592689 4905 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b38f224-a70d-4605-a35b-c4ae45ff122a-webhook-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:52.592749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.592698 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzr2q\" (UniqueName: \"kubernetes.io/projected/5b38f224-a70d-4605-a35b-c4ae45ff122a-kube-api-access-tzr2q\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:28:52.632902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.632875 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-jn4xq"] Oct 30 01:28:52.636072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:52.636048 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-jn4xq"] Oct 30 01:28:53.190339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.190304 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b38f224-a70d-4605-a35b-c4ae45ff122a" path="/var/lib/kubelet/pods/5b38f224-a70d-4605-a35b-c4ae45ff122a/volumes" Oct 30 01:28:53.290632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.290602 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-b79zr"] Oct 30 01:28:53.291002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.290990 4905 cpu_manager.go:414] "containerMap: removing container" podUID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" containerName="payload-processing" Oct 30 01:28:53.291056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291004 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" containerName="payload-processing" Oct 30 01:28:53.291056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291018 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" containerName="payload-pre-processing" Oct 30 01:28:53.291056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291024 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" containerName="payload-pre-processing" Oct 30 01:28:53.291056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291033 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5af47d61-5943-430a-9a1b-a8f7f12a952a" containerName="revoke-keys" Oct 30 01:28:53.291056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291038 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af47d61-5943-430a-9a1b-a8f7f12a952a" containerName="revoke-keys" Oct 30 01:28:53.291056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291047 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" Oct 30 01:28:53.291056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291052 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291061 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5b38f224-a70d-4605-a35b-c4ae45ff122a" containerName="manager" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291066 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b38f224-a70d-4605-a35b-c4ae45ff122a" containerName="manager" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291082 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291087 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291139 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab69f74-0877-4f68-a0dd-b4a631fead70" containerName="main" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291151 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff91ee3-305c-4562-a05b-4eac7828ef1c" containerName="payload-processing" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291160 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd0d9fb-f75a-4f0a-b984-1f64ad207014" containerName="payload-pre-processing" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291166 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af47d61-5943-430a-9a1b-a8f7f12a952a" containerName="revoke-keys" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291173 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d3906c-bd1a-4817-afc3-bb8158310757" containerName="main" Oct 30 01:28:53.291308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.291178 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b38f224-a70d-4605-a35b-c4ae45ff122a" containerName="manager" Oct 30 01:28:53.295632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.295611 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.311419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.311399 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 01:28:53.325175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.325152 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w6xmx\"" Oct 30 01:28:53.325358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.325152 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 01:28:53.325851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.325824 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 01:28:53.327412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.327390 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-b79zr"] Oct 30 01:28:53.400841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.400809 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46r5h\" (UniqueName: \"kubernetes.io/projected/55402ccc-92d6-47e6-98db-fd8d327c96ef-kube-api-access-46r5h\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.401022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.400845 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-metrics-certs\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.401022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.400970 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-webhook-cert\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.501789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.501748 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-metrics-certs\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.501986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.501857 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-webhook-cert\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.502066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.501989 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46r5h\" (UniqueName: \"kubernetes.io/projected/55402ccc-92d6-47e6-98db-fd8d327c96ef-kube-api-access-46r5h\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.504411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.504382 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-metrics-certs\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.504504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.504427 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-webhook-cert\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.521994 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.521968 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46r5h\" (UniqueName: \"kubernetes.io/projected/55402ccc-92d6-47e6-98db-fd8d327c96ef-kube-api-access-46r5h\") pod \"maas-controller-78d6f77c9d-b79zr\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.608722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.608690 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:53.907997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:53.907966 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-b79zr"] Oct 30 01:28:53.908913 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:28:53.908887 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55402ccc_92d6_47e6_98db_fd8d327c96ef.slice/crio-fd0613991f4c58f48b58ddb9573c238e726f38bf5a2579db524978323ac72976 WatchSource:0}: Error finding container fd0613991f4c58f48b58ddb9573c238e726f38bf5a2579db524978323ac72976: Status 404 returned error can't find the container with id fd0613991f4c58f48b58ddb9573c238e726f38bf5a2579db524978323ac72976 Oct 30 01:28:54.590808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:54.590721 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" event={"ID":"55402ccc-92d6-47e6-98db-fd8d327c96ef","Type":"ContainerStarted","Data":"9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d"} Oct 30 01:28:54.590808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:54.590785 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" event={"ID":"55402ccc-92d6-47e6-98db-fd8d327c96ef","Type":"ContainerStarted","Data":"fd0613991f4c58f48b58ddb9573c238e726f38bf5a2579db524978323ac72976"} Oct 30 01:28:54.591301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:54.590936 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:28:54.633740 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:28:54.633689 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" podStartSLOduration=1.2465586339999999 podStartE2EDuration="1.633675972s" totalImagesPullingTime="387.117338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:54.631772881 +0000 UTC m=+3019.574946073" watchObservedRunningTime="2026-10-30 01:28:54.633675972 +0000 UTC m=+3019.576849162" Oct 30 01:29:05.599050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:05.599019 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:29:12.414092 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:29:12.414053 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:29:15.601372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.601339 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt"] Oct 30 01:29:15.603775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.603760 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.633872 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.633844 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:29:15.635050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.635024 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:29:15.635189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.635092 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:29:15.635262 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.635197 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:29:15.636790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.636757 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt"] Oct 30 01:29:15.697658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.697625 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmncv\" (UniqueName: \"kubernetes.io/projected/0f95758d-68d4-44cc-84d9-a8a13e1a363d-kube-api-access-mmncv\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.697834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.697736 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.697834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.697763 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.798489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.798455 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmncv\" (UniqueName: \"kubernetes.io/projected/0f95758d-68d4-44cc-84d9-a8a13e1a363d-kube-api-access-mmncv\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.798696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.798552 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.798696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.798572 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.799274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.799254 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.801004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.800985 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.818780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.818753 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmncv\" (UniqueName: \"kubernetes.io/projected/0f95758d-68d4-44cc-84d9-a8a13e1a363d-kube-api-access-mmncv\") pod \"maas-api-revoke-sub-85237e45-ml9pt\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:15.917228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:15.917155 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:16.213379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:16.213255 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt"] Oct 30 01:29:16.701151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:16.701108 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" event={"ID":"0f95758d-68d4-44cc-84d9-a8a13e1a363d","Type":"ContainerStarted","Data":"7701a3d30d5ef2fee0dbe69378e6d068bc3f6ea3900cbea2f004ad3800910bec"} Oct 30 01:29:16.701683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:16.701177 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" event={"ID":"0f95758d-68d4-44cc-84d9-a8a13e1a363d","Type":"ContainerStarted","Data":"13f5edd48ba81c1ef4ee4e1220e845259188168e88531e340a9ccb22a9066d77"} Oct 30 01:29:16.743567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:16.743499 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" podStartSLOduration=1.743486696 podStartE2EDuration="1.743486696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:16.742412495 +0000 UTC m=+3041.685585689" watchObservedRunningTime="2026-10-30 01:29:16.743486696 +0000 UTC m=+3041.686659882" Oct 30 01:29:17.708348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:17.708259 4905 generic.go:356] "Generic (PLEG): container finished" podID="0f95758d-68d4-44cc-84d9-a8a13e1a363d" containerID="7701a3d30d5ef2fee0dbe69378e6d068bc3f6ea3900cbea2f004ad3800910bec" exitCode=0 Oct 30 01:29:17.708815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:17.708336 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" event={"ID":"0f95758d-68d4-44cc-84d9-a8a13e1a363d","Type":"ContainerDied","Data":"7701a3d30d5ef2fee0dbe69378e6d068bc3f6ea3900cbea2f004ad3800910bec"} Oct 30 01:29:18.847858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:18.847837 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:18.929758 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:18.929731 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-xks-ca\") pod \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " Oct 30 01:29:18.929911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:18.929830 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-openshift-ca\") pod \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " Oct 30 01:29:18.929911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:18.929854 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f95758d-68d4-44cc-84d9-a8a13e1a363d-kube-api-access-mmncv\" (UniqueName: \"kubernetes.io/projected/0f95758d-68d4-44cc-84d9-a8a13e1a363d-kube-api-access-mmncv\") pod \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\" (UID: \"0f95758d-68d4-44cc-84d9-a8a13e1a363d\") " Oct 30 01:29:18.930226 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:18.930199 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-openshift-ca" pod "0f95758d-68d4-44cc-84d9-a8a13e1a363d" (UID: "0f95758d-68d4-44cc-84d9-a8a13e1a363d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:18.931940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:18.931917 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-xks-ca" pod "0f95758d-68d4-44cc-84d9-a8a13e1a363d" (UID: "0f95758d-68d4-44cc-84d9-a8a13e1a363d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:18.932069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:18.932055 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f95758d-68d4-44cc-84d9-a8a13e1a363d-kube-api-access-mmncv" pod "0f95758d-68d4-44cc-84d9-a8a13e1a363d" (UID: "0f95758d-68d4-44cc-84d9-a8a13e1a363d"). InnerVolumeSpecName "kube-api-access-mmncv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:19.031453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:19.031426 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:29:19.031453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:19.031450 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmncv\" (UniqueName: \"kubernetes.io/projected/0f95758d-68d4-44cc-84d9-a8a13e1a363d-kube-api-access-mmncv\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:29:19.031649 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:19.031460 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95758d-68d4-44cc-84d9-a8a13e1a363d-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:29:19.719607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:19.719570 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" event={"ID":"0f95758d-68d4-44cc-84d9-a8a13e1a363d","Type":"ContainerDied","Data":"13f5edd48ba81c1ef4ee4e1220e845259188168e88531e340a9ccb22a9066d77"} Oct 30 01:29:19.719781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:19.719619 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13f5edd48ba81c1ef4ee4e1220e845259188168e88531e340a9ccb22a9066d77" Oct 30 01:29:19.719781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:19.719587 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt" Oct 30 01:29:20.776703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.776669 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2"] Oct 30 01:29:20.777125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.777080 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0f95758d-68d4-44cc-84d9-a8a13e1a363d" containerName="revoke-keys" Oct 30 01:29:20.777125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.777090 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f95758d-68d4-44cc-84d9-a8a13e1a363d" containerName="revoke-keys" Oct 30 01:29:20.777213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.777156 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f95758d-68d4-44cc-84d9-a8a13e1a363d" containerName="revoke-keys" Oct 30 01:29:20.779158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.779136 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.790543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.790493 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs"] Oct 30 01:29:20.792943 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.792929 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.802504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.802484 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rxg2r\"" Oct 30 01:29:20.807843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.807811 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2"] Oct 30 01:29:20.825601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.825575 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs"] Oct 30 01:29:20.847759 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.847731 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.847874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.847771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q969q\" (UniqueName: \"kubernetes.io/projected/e5fe03d5-2065-4957-a660-22bd064b5892-kube-api-access-q969q\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.847874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.847844 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.847969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.847876 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fe03d5-2065-4957-a660-22bd064b5892-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.848017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.847970 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.848017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.847995 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fe03d5-2065-4957-a660-22bd064b5892-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.848091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.848047 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.848091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.848079 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.848211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.848108 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fe03d5-2065-4957-a660-22bd064b5892-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.949564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949506 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.949713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949588 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q969q\" (UniqueName: \"kubernetes.io/projected/e5fe03d5-2065-4957-a660-22bd064b5892-kube-api-access-q969q\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.949713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949623 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.949713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949654 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.949847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949827 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.949902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949861 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.949902 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949888 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b68d31-1fc7-4045-9167-11e44e372333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.950011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949920 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fe03d5-2065-4957-a660-22bd064b5892-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950011 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.949937 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950085 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b68d31-1fc7-4045-9167-11e44e372333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.950133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950129 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950176 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fe03d5-2065-4957-a660-22bd064b5892-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950195 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950174 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950216 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950284 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fe03d5-2065-4957-a660-22bd064b5892-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950317 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.950487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950345 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.950487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950397 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950432 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq94h\" (UniqueName: \"kubernetes.io/projected/f9b68d31-1fc7-4045-9167-11e44e372333-kube-api-access-pq94h\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.950826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950488 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b68d31-1fc7-4045-9167-11e44e372333-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:20.950826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950565 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.950935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.950871 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fe03d5-2065-4957-a660-22bd064b5892-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.952427 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.952400 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fe03d5-2065-4957-a660-22bd064b5892-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.952588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.952552 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fe03d5-2065-4957-a660-22bd064b5892-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.969701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.969673 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q969q\" (UniqueName: \"kubernetes.io/projected/e5fe03d5-2065-4957-a660-22bd064b5892-kube-api-access-q969q\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:20.969948 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:20.969925 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fe03d5-2065-4957-a660-22bd064b5892-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pwsj2\" (UID: \"e5fe03d5-2065-4957-a660-22bd064b5892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:21.051410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051339 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b68d31-1fc7-4045-9167-11e44e372333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051443 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051471 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051496 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq94h\" (UniqueName: \"kubernetes.io/projected/f9b68d31-1fc7-4045-9167-11e44e372333-kube-api-access-pq94h\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051548 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b68d31-1fc7-4045-9167-11e44e372333-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051608 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051675 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051719 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.051843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051755 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b68d31-1fc7-4045-9167-11e44e372333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.052079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.051992 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.052079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.052026 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.052166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.052078 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.052213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.052188 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.052754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.052729 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b68d31-1fc7-4045-9167-11e44e372333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.053984 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.053962 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b68d31-1fc7-4045-9167-11e44e372333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.054081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.054062 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b68d31-1fc7-4045-9167-11e44e372333-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.072117 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.072095 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b68d31-1fc7-4045-9167-11e44e372333-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.072202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.072167 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq94h\" (UniqueName: \"kubernetes.io/projected/f9b68d31-1fc7-4045-9167-11e44e372333-kube-api-access-pq94h\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hzfrs\" (UID: \"f9b68d31-1fc7-4045-9167-11e44e372333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.091966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.091946 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:21.103761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.103741 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:21.411907 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:29:21.411866 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b68d31_1fc7_4045_9167_11e44e372333.slice/crio-d7b032df5e48da81b75abbcb44af01db02d95fbdcb0a3c1d2833ac280203aa22 WatchSource:0}: Error finding container d7b032df5e48da81b75abbcb44af01db02d95fbdcb0a3c1d2833ac280203aa22: Status 404 returned error can't find the container with id d7b032df5e48da81b75abbcb44af01db02d95fbdcb0a3c1d2833ac280203aa22 Oct 30 01:29:21.412085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.412060 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2"] Oct 30 01:29:21.412506 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:29:21.412483 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fe03d5_2065_4957_a660_22bd064b5892.slice/crio-6e1a82b292b478dda6fc8104c64733d10d3599a6d6254712357db44444cf6564 WatchSource:0}: Error finding container 6e1a82b292b478dda6fc8104c64733d10d3599a6d6254712357db44444cf6564: Status 404 returned error can't find the container with id 6e1a82b292b478dda6fc8104c64733d10d3599a6d6254712357db44444cf6564 Oct 30 01:29:21.414071 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.414048 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs"] Oct 30 01:29:21.414166 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.414117 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:21.414249 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.414223 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:21.414293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.414271 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:21.414782 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.414750 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:21.414825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.414802 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:21.414860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.414832 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:21.734617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.734577 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" event={"ID":"e5fe03d5-2065-4957-a660-22bd064b5892","Type":"ContainerStarted","Data":"65dba4e8c1ba5b437d6859310cb3bbca5e9c1385de4ef098321ba012f80e9c67"} Oct 30 01:29:21.734809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.734650 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" event={"ID":"e5fe03d5-2065-4957-a660-22bd064b5892","Type":"ContainerStarted","Data":"6e1a82b292b478dda6fc8104c64733d10d3599a6d6254712357db44444cf6564"} Oct 30 01:29:21.736750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.736720 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" event={"ID":"f9b68d31-1fc7-4045-9167-11e44e372333","Type":"ContainerStarted","Data":"759e3ec5ff22a45ba509d17f44d00c7983b1a7e9406a0c7255bf3da7ce9dfb20"} Oct 30 01:29:21.736876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.736775 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" event={"ID":"f9b68d31-1fc7-4045-9167-11e44e372333","Type":"ContainerStarted","Data":"d7b032df5e48da81b75abbcb44af01db02d95fbdcb0a3c1d2833ac280203aa22"} Oct 30 01:29:21.781196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.781137 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podStartSLOduration=1.781122698 podStartE2EDuration="1.781122698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:21.778771213 +0000 UTC m=+3046.721944406" watchObservedRunningTime="2026-10-30 01:29:21.781122698 +0000 UTC m=+3046.724295890" Oct 30 01:29:21.824301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:21.824252 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podStartSLOduration=1.8242353329999998 podStartE2EDuration="1.824235333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:21.822725428 +0000 UTC m=+3046.765898634" watchObservedRunningTime="2026-10-30 01:29:21.824235333 +0000 UTC m=+3046.767408525" Oct 30 01:29:22.092440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:22.092361 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:22.093703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:22.093681 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 01:29:22.093843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:22.093717 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 01:29:22.104569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:22.104546 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:22.105464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:22.105443 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 30 01:29:22.105578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:22.105484 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 30 01:29:23.092555 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:23.092500 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 01:29:23.092978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:23.092571 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 01:29:23.105332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:23.105302 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 30 01:29:23.105468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:23.105350 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 30 01:29:24.092973 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:24.092937 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 01:29:24.093381 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:24.092988 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 01:29:24.104678 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:24.104643 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 30 01:29:24.104833 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:24.104700 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 30 01:29:25.092279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:25.092238 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 01:29:25.092507 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:25.092288 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 01:29:25.104061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:25.104025 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 30 01:29:25.104435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:25.104082 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 30 01:29:26.093356 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:26.093315 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 01:29:26.093595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:26.093367 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 01:29:26.104946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:26.104921 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 30 01:29:26.105270 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:26.104969 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 30 01:29:27.092860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.092826 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 01:29:27.093024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.092889 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 01:29:27.104587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.104555 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 30 01:29:27.104725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.104603 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 30 01:29:27.418332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.418232 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-b79zr"] Oct 30 01:29:27.418847 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.418491 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" podUID="55402ccc-92d6-47e6-98db-fd8d327c96ef" containerName="manager" containerID="cri-o://9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d" gracePeriod=30 Oct 30 01:29:27.680769 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.680707 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:29:27.805089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.805052 4905 generic.go:356] "Generic (PLEG): container finished" podID="55402ccc-92d6-47e6-98db-fd8d327c96ef" containerID="9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d" exitCode=0 Oct 30 01:29:27.805286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.805112 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" Oct 30 01:29:27.805286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.805123 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" event={"ID":"55402ccc-92d6-47e6-98db-fd8d327c96ef","Type":"ContainerDied","Data":"9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d"} Oct 30 01:29:27.805286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.805191 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-b79zr" event={"ID":"55402ccc-92d6-47e6-98db-fd8d327c96ef","Type":"ContainerDied","Data":"fd0613991f4c58f48b58ddb9573c238e726f38bf5a2579db524978323ac72976"} Oct 30 01:29:27.805286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.805224 4905 scope.go:122] "RemoveContainer" containerID="9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d" Oct 30 01:29:27.815648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.815626 4905 scope.go:122] "RemoveContainer" containerID="9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d" Oct 30 01:29:27.815968 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:29:27.815938 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d\": container with ID starting with 9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d not found: ID does not exist" containerID="9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d" Oct 30 01:29:27.816105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.815980 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d"} err="failed to get container status \"9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d\": rpc error: code = NotFound desc = could not find container \"9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d\": container with ID starting with 9a171e2db4a4f64ae64672cfd88434103b161082cbc787054c51f0dd58a0868d not found: ID does not exist" Oct 30 01:29:27.826501 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.826461 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55402ccc-92d6-47e6-98db-fd8d327c96ef-kube-api-access-46r5h\" (UniqueName: \"kubernetes.io/projected/55402ccc-92d6-47e6-98db-fd8d327c96ef-kube-api-access-46r5h\") pod \"55402ccc-92d6-47e6-98db-fd8d327c96ef\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " Oct 30 01:29:27.826647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.826625 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-metrics-certs\") pod \"55402ccc-92d6-47e6-98db-fd8d327c96ef\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " Oct 30 01:29:27.826730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.826655 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-webhook-cert\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-webhook-cert\") pod \"55402ccc-92d6-47e6-98db-fd8d327c96ef\" (UID: \"55402ccc-92d6-47e6-98db-fd8d327c96ef\") " Oct 30 01:29:27.828903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.828880 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-webhook-cert" pod "55402ccc-92d6-47e6-98db-fd8d327c96ef" (UID: "55402ccc-92d6-47e6-98db-fd8d327c96ef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:27.829348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.829322 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-metrics-certs" pod "55402ccc-92d6-47e6-98db-fd8d327c96ef" (UID: "55402ccc-92d6-47e6-98db-fd8d327c96ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:27.830961 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.830937 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55402ccc-92d6-47e6-98db-fd8d327c96ef-kube-api-access-46r5h" pod "55402ccc-92d6-47e6-98db-fd8d327c96ef" (UID: "55402ccc-92d6-47e6-98db-fd8d327c96ef"). InnerVolumeSpecName "kube-api-access-46r5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:27.927609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.927577 4905 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-webhook-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:29:27.927609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.927606 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46r5h\" (UniqueName: \"kubernetes.io/projected/55402ccc-92d6-47e6-98db-fd8d327c96ef-kube-api-access-46r5h\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:29:27.927793 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:27.927616 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55402ccc-92d6-47e6-98db-fd8d327c96ef-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:29:28.092313 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:28.092277 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 01:29:28.092504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:28.092333 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 01:29:28.104113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:28.104084 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 30 01:29:28.104215 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:28.104137 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 30 01:29:28.155491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:28.155459 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-b79zr"] Oct 30 01:29:28.161316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:28.161291 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-b79zr"] Oct 30 01:29:29.193079 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.193032 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55402ccc-92d6-47e6-98db-fd8d327c96ef" path="/var/lib/kubelet/pods/55402ccc-92d6-47e6-98db-fd8d327c96ef/volumes" Oct 30 01:29:29.354121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.354083 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-7kkl4"] Oct 30 01:29:29.354842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.354811 4905 cpu_manager.go:414] "containerMap: removing container" podUID="55402ccc-92d6-47e6-98db-fd8d327c96ef" containerName="manager" Oct 30 01:29:29.354842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.354840 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55402ccc-92d6-47e6-98db-fd8d327c96ef" containerName="manager" Oct 30 01:29:29.355044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.354962 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="55402ccc-92d6-47e6-98db-fd8d327c96ef" containerName="manager" Oct 30 01:29:29.359630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.359604 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.376018 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.375991 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 01:29:29.391552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.389646 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 01:29:29.391552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.389763 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w6xmx\"" Oct 30 01:29:29.391552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.389832 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 01:29:29.393791 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.393762 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-7kkl4"] Oct 30 01:29:29.445082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.444999 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/972beb40-dba6-4ac0-9848-94188dadbe68-webhook-cert\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.445233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.445160 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfz7\" (UniqueName: \"kubernetes.io/projected/972beb40-dba6-4ac0-9848-94188dadbe68-kube-api-access-kgfz7\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.445233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.445213 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972beb40-dba6-4ac0-9848-94188dadbe68-metrics-certs\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.546271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.546224 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfz7\" (UniqueName: \"kubernetes.io/projected/972beb40-dba6-4ac0-9848-94188dadbe68-kube-api-access-kgfz7\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.546481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.546288 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972beb40-dba6-4ac0-9848-94188dadbe68-metrics-certs\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.546481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.546390 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/972beb40-dba6-4ac0-9848-94188dadbe68-webhook-cert\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.549849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.549815 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/972beb40-dba6-4ac0-9848-94188dadbe68-webhook-cert\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.549965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.549851 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972beb40-dba6-4ac0-9848-94188dadbe68-metrics-certs\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.569846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.569813 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfz7\" (UniqueName: \"kubernetes.io/projected/972beb40-dba6-4ac0-9848-94188dadbe68-kube-api-access-kgfz7\") pod \"maas-controller-78d6f77c9d-7kkl4\" (UID: \"972beb40-dba6-4ac0-9848-94188dadbe68\") " pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.678218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.678160 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:29.786604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.786575 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:29.787631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.787606 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:29.788479 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.788460 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" Oct 30 01:29:29.805199 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.805177 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:29.805795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.805637 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:29.806317 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.806292 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" Oct 30 01:29:29.998191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:29.998162 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d6f77c9d-7kkl4"] Oct 30 01:29:29.998367 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:29:29.998346 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972beb40_dba6_4ac0_9848_94188dadbe68.slice/crio-653d6e88f374e44af2213b2861d25cbe0241866454958d7ae4639af2d87c2b02 WatchSource:0}: Error finding container 653d6e88f374e44af2213b2861d25cbe0241866454958d7ae4639af2d87c2b02: Status 404 returned error can't find the container with id 653d6e88f374e44af2213b2861d25cbe0241866454958d7ae4639af2d87c2b02 Oct 30 01:29:30.824625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:30.824583 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" event={"ID":"972beb40-dba6-4ac0-9848-94188dadbe68","Type":"ContainerStarted","Data":"d64b3e52dadd8f9b12f2ff8810bbbb1223740bc80912e3f9438bbb2fd3f5d616"} Oct 30 01:29:30.825083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:30.824649 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" event={"ID":"972beb40-dba6-4ac0-9848-94188dadbe68","Type":"ContainerStarted","Data":"653d6e88f374e44af2213b2861d25cbe0241866454958d7ae4639af2d87c2b02"} Oct 30 01:29:30.868167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:30.868118 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" podStartSLOduration=1.512074446 podStartE2EDuration="1.868106128s" totalImagesPullingTime="356.031682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:30.86657885 +0000 UTC m=+3055.809752043" watchObservedRunningTime="2026-10-30 01:29:30.868106128 +0000 UTC m=+3055.811279315" Oct 30 01:29:31.831322 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:31.831292 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:35.776395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.776361 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m"] Oct 30 01:29:35.780149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.780126 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.818243 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.814843 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m"] Oct 30 01:29:35.911239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911201 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911244 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911264 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e721927-370e-4f47-9eef-60330ef5886a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911369 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e721927-370e-4f47-9eef-60330ef5886a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911418 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911470 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911514 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e721927-370e-4f47-9eef-60330ef5886a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911566 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:35.911654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:35.911640 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2lz\" (UniqueName: \"kubernetes.io/projected/2e721927-370e-4f47-9eef-60330ef5886a-kube-api-access-jz2lz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.012787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.012745 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.012787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.012796 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.012998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.012815 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e721927-370e-4f47-9eef-60330ef5886a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.012998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.012842 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e721927-370e-4f47-9eef-60330ef5886a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.012998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.012903 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.012998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.012932 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.012998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.012976 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e721927-370e-4f47-9eef-60330ef5886a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.013280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.013014 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.013280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.013040 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2lz\" (UniqueName: \"kubernetes.io/projected/2e721927-370e-4f47-9eef-60330ef5886a-kube-api-access-jz2lz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.013280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.013254 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.013448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.013278 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.013448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.013409 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.013625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.013499 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.013767 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.013744 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e721927-370e-4f47-9eef-60330ef5886a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.015263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.015240 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e721927-370e-4f47-9eef-60330ef5886a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.015487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.015467 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e721927-370e-4f47-9eef-60330ef5886a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.033234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.033175 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e721927-370e-4f47-9eef-60330ef5886a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.033318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.033238 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2lz\" (UniqueName: \"kubernetes.io/projected/2e721927-370e-4f47-9eef-60330ef5886a-kube-api-access-jz2lz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2rs7m\" (UID: \"2e721927-370e-4f47-9eef-60330ef5886a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.094259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.094237 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:36.401336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.401309 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m"] Oct 30 01:29:36.401798 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:29:36.401775 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e721927_370e_4f47_9eef_60330ef5886a.slice/crio-37d7429aa732816d7d8b20d0af4e3a95a8be98e91261b947b349018646883052 WatchSource:0}: Error finding container 37d7429aa732816d7d8b20d0af4e3a95a8be98e91261b947b349018646883052: Status 404 returned error can't find the container with id 37d7429aa732816d7d8b20d0af4e3a95a8be98e91261b947b349018646883052 Oct 30 01:29:36.404002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.403966 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:36.404112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.404041 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:36.404112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.404088 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:29:36.860035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.859994 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" event={"ID":"2e721927-370e-4f47-9eef-60330ef5886a","Type":"ContainerStarted","Data":"f4bb56baa957d529bf885833db9ec651c1bd91f0fc7bb9f2f3da300ad29fe072"} Oct 30 01:29:36.860601 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.860059 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" event={"ID":"2e721927-370e-4f47-9eef-60330ef5886a","Type":"ContainerStarted","Data":"37d7429aa732816d7d8b20d0af4e3a95a8be98e91261b947b349018646883052"} Oct 30 01:29:36.923305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:36.923248 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podStartSLOduration=1.923231757 podStartE2EDuration="1.923231757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:36.921576634 +0000 UTC m=+3061.864749828" watchObservedRunningTime="2026-10-30 01:29:36.923231757 +0000 UTC m=+3061.866404952" Oct 30 01:29:37.095438 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:37.095410 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:37.096662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:37.096637 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 01:29:37.096801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:37.096684 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 01:29:38.094655 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:38.094620 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 01:29:38.095137 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:38.094673 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 01:29:39.095405 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:39.095370 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 01:29:39.095835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:39.095421 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 01:29:40.095362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:40.095330 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 01:29:40.095580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:40.095384 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 01:29:41.094571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:41.094531 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 01:29:41.094765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:41.094598 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 01:29:42.095354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:42.095320 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 01:29:42.095790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:42.095375 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 01:29:42.414074 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:29:42.413980 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:29:42.840246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:42.840213 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78d6f77c9d-7kkl4" Oct 30 01:29:43.094750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:43.094664 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 01:29:43.094750 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:43.094723 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 01:29:44.619112 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:44.619080 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:44.619584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:44.619557 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:29:44.620460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:29:44.620441 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" Oct 30 01:30:00.149185 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.149145 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz"] Oct 30 01:30:00.212393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.212358 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz"] Oct 30 01:30:00.212830 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.212737 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.245999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.245906 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:00.246161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.246045 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:30:00.246161 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.246138 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:00.321153 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.321109 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5rr\" (UniqueName: \"kubernetes.io/projected/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-kube-api-access-4j5rr\") pod \"maas-api-key-cleanup-29888730-phglz\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.321493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.321464 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-phglz\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.422868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.422789 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5rr\" (UniqueName: \"kubernetes.io/projected/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-kube-api-access-4j5rr\") pod \"maas-api-key-cleanup-29888730-phglz\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.422868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.422863 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-phglz\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.423498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.423479 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-phglz\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.461751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.461717 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5rr\" (UniqueName: \"kubernetes.io/projected/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-kube-api-access-4j5rr\") pod \"maas-api-key-cleanup-29888730-phglz\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.595796 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.595749 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:00.843827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.843401 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hzfrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:30:00.843827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.843475 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hzfrs" podUID="f9b68d31-1fc7-4045-9167-11e44e372333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:30:00.911749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.850864 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:30:00.911749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:00.850913 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:30:01.117574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:01.117475 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz"] Oct 30 01:30:02.026183 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:02.026139 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" event={"ID":"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b","Type":"ContainerStarted","Data":"09500a46ea9010f13d3880fb907a7d7eb856895c95bf1383b4c24169000414dd"} Oct 30 01:30:02.026621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:02.026210 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" event={"ID":"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b","Type":"ContainerStarted","Data":"64144e482e223ddd56883618bd8c2c449c091a2aee5f6649c309169441d65bb8"} Oct 30 01:30:02.068696 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:02.068629 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" podStartSLOduration=2.068609684 podStartE2EDuration="2.068609684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.067795221 +0000 UTC m=+3087.010968415" watchObservedRunningTime="2026-10-30 01:30:02.068609684 +0000 UTC m=+3087.011782921" Oct 30 01:30:03.033602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:03.033564 4905 generic.go:356] "Generic (PLEG): container finished" podID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerID="09500a46ea9010f13d3880fb907a7d7eb856895c95bf1383b4c24169000414dd" exitCode=60 Oct 30 01:30:03.034029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:03.033642 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" event={"ID":"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b","Type":"ContainerDied","Data":"09500a46ea9010f13d3880fb907a7d7eb856895c95bf1383b4c24169000414dd"} Oct 30 01:30:03.034029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:03.033872 4905 scope.go:122] "RemoveContainer" containerID="09500a46ea9010f13d3880fb907a7d7eb856895c95bf1383b4c24169000414dd" Oct 30 01:30:04.040814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:04.040781 4905 generic.go:356] "Generic (PLEG): container finished" podID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerID="a8f4777659811108b99fec703d72251ec22037cd1574e1661b92f2c1e411e7f4" exitCode=60 Oct 30 01:30:04.041237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:04.040877 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" event={"ID":"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b","Type":"ContainerDied","Data":"a8f4777659811108b99fec703d72251ec22037cd1574e1661b92f2c1e411e7f4"} Oct 30 01:30:04.041237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:04.040937 4905 scope.go:122] "RemoveContainer" containerID="09500a46ea9010f13d3880fb907a7d7eb856895c95bf1383b4c24169000414dd" Oct 30 01:30:04.041237 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:04.041214 4905 scope.go:122] "RemoveContainer" containerID="a8f4777659811108b99fec703d72251ec22037cd1574e1661b92f2c1e411e7f4" Oct 30 01:30:04.041483 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:30:04.041463 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-phglz_odh-ai-gateway-infra(bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" Oct 30 01:30:05.055775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:05.055745 4905 scope.go:122] "RemoveContainer" containerID="a8f4777659811108b99fec703d72251ec22037cd1574e1661b92f2c1e411e7f4" Oct 30 01:30:05.056259 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:30:05.055961 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-phglz_odh-ai-gateway-infra(bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" Oct 30 01:30:12.414172 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:30:12.414135 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:30:18.184359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:18.184164 4905 scope.go:122] "RemoveContainer" containerID="a8f4777659811108b99fec703d72251ec22037cd1574e1661b92f2c1e411e7f4" Oct 30 01:30:19.143334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:19.143304 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" event={"ID":"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b","Type":"ContainerStarted","Data":"dddb7a5dd11337cb837ecb73c09809052aaa6070b642a293443ce24dab82463c"} Oct 30 01:30:20.151339 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.151305 4905 generic.go:356] "Generic (PLEG): container finished" podID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerID="dddb7a5dd11337cb837ecb73c09809052aaa6070b642a293443ce24dab82463c" exitCode=60 Oct 30 01:30:20.151804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.151380 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" event={"ID":"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b","Type":"ContainerDied","Data":"dddb7a5dd11337cb837ecb73c09809052aaa6070b642a293443ce24dab82463c"} Oct 30 01:30:20.151804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.151437 4905 scope.go:122] "RemoveContainer" containerID="a8f4777659811108b99fec703d72251ec22037cd1574e1661b92f2c1e411e7f4" Oct 30 01:30:20.151919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.151825 4905 scope.go:122] "RemoveContainer" containerID="dddb7a5dd11337cb837ecb73c09809052aaa6070b642a293443ce24dab82463c" Oct 30 01:30:20.152095 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:30:20.152074 4905 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-phglz_odh-ai-gateway-infra(bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" Oct 30 01:30:20.193794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.193766 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz"] Oct 30 01:30:20.796049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.796014 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245"] Oct 30 01:30:20.800799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.800781 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.822026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.821982 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245"] Oct 30 01:30:20.904223 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904189 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1988474-6586-4686-b98d-34a146461f5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904250 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904319 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904340 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1988474-6586-4686-b98d-34a146461f5e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904433 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904367 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904438 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904499 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904591 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1988474-6586-4686-b98d-34a146461f5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:20.904721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:20.904628 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvlw\" (UniqueName: \"kubernetes.io/projected/b1988474-6586-4686-b98d-34a146461f5e-kube-api-access-fnvlw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005234 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005197 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1988474-6586-4686-b98d-34a146461f5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005264 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005294 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005403 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1988474-6586-4686-b98d-34a146461f5e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005472 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005442 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005491 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005771 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005534 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005917 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1988474-6586-4686-b98d-34a146461f5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.005977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.005956 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvlw\" (UniqueName: \"kubernetes.io/projected/b1988474-6586-4686-b98d-34a146461f5e-kube-api-access-fnvlw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.006164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.006052 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1988474-6586-4686-b98d-34a146461f5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.006164 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.006077 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.006605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.006581 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.006785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.006631 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.008445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.008417 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1988474-6586-4686-b98d-34a146461f5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.009081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.009053 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1988474-6586-4686-b98d-34a146461f5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.026721 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.026695 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1988474-6586-4686-b98d-34a146461f5e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.027121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.027096 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvlw\" (UniqueName: \"kubernetes.io/projected/b1988474-6586-4686-b98d-34a146461f5e-kube-api-access-fnvlw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2z245\" (UID: \"b1988474-6586-4686-b98d-34a146461f5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.115004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.114900 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:21.391483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.391449 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:21.485701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.485662 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245"] Oct 30 01:30:21.491211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.490475 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:30:21.491211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.490585 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:30:21.491211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.490636 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:30:21.515802 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.515777 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-kube-api-access-4j5rr\" (UniqueName: \"kubernetes.io/projected/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-kube-api-access-4j5rr\") pod \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " Oct 30 01:30:21.515944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.515823 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-maas-api-service-ca\") pod \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\" (UID: \"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b\") " Oct 30 01:30:21.516302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.516270 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-maas-api-service-ca" pod "bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" (UID: "bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:21.519471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.518378 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-kube-api-access-4j5rr" pod "bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" (UID: "bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b"). InnerVolumeSpecName "kube-api-access-4j5rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:21.617459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.617419 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5rr\" (UniqueName: \"kubernetes.io/projected/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-kube-api-access-4j5rr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:21.617459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:21.617460 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b-maas-api-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:22.190545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.189051 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" event={"ID":"b1988474-6586-4686-b98d-34a146461f5e","Type":"ContainerStarted","Data":"4491ccc3e22396dc538cd9b7074327a9586b47ab740b2ec326b28f0b03ae6ecf"} Oct 30 01:30:22.190545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.189124 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" event={"ID":"b1988474-6586-4686-b98d-34a146461f5e","Type":"ContainerStarted","Data":"11e2b2bc7b34e5f1f53a46f13d2c841eceafa8c569e2b1ffdc4ee0512143459c"} Oct 30 01:30:22.194541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.192045 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" event={"ID":"bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b","Type":"ContainerDied","Data":"64144e482e223ddd56883618bd8c2c449c091a2aee5f6649c309169441d65bb8"} Oct 30 01:30:22.194541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.192107 4905 scope.go:122] "RemoveContainer" containerID="dddb7a5dd11337cb837ecb73c09809052aaa6070b642a293443ce24dab82463c" Oct 30 01:30:22.194541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.192199 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz" Oct 30 01:30:22.245560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.243440 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podStartSLOduration=2.243420883 podStartE2EDuration="2.243420883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:22.240882714 +0000 UTC m=+3107.184055911" watchObservedRunningTime="2026-10-30 01:30:22.243420883 +0000 UTC m=+3107.186594077" Oct 30 01:30:22.373089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.372501 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz"] Oct 30 01:30:22.419545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:22.417003 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-phglz"] Oct 30 01:30:23.124547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:23.124008 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:23.149867 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:23.149827 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2z245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 30 01:30:23.150154 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:23.150122 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podUID="b1988474-6586-4686-b98d-34a146461f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 30 01:30:23.231546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:23.230807 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" path="/var/lib/kubelet/pods/bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b/volumes" Oct 30 01:30:24.115764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:24.115733 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2z245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 30 01:30:24.116201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:24.115788 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podUID="b1988474-6586-4686-b98d-34a146461f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 30 01:30:25.115470 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:25.115435 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2z245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 30 01:30:25.115678 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:25.115490 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podUID="b1988474-6586-4686-b98d-34a146461f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 30 01:30:26.116088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:26.116051 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2z245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 30 01:30:26.116493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:26.116111 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podUID="b1988474-6586-4686-b98d-34a146461f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 30 01:30:27.115805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.115770 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2z245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 30 01:30:27.115999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.115824 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podUID="b1988474-6586-4686-b98d-34a146461f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 30 01:30:27.646659 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.646620 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf"] Oct 30 01:30:27.648221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.648194 4905 cpu_manager.go:414] "containerMap: removing container" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.650066 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.650040 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.650193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.650085 4905 cpu_manager.go:414] "containerMap: removing container" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.650193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.650095 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.650193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.650121 4905 cpu_manager.go:414] "containerMap: removing container" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.650193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.650129 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.650384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.650280 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.650384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.650298 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:27.655267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.655242 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf"] Oct 30 01:30:27.655360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.655339 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.671630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.671609 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:27.685469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.685439 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:27.685605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.685550 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:27.685605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.685581 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:30:27.759674 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.759631 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543ea719-e826-47dd-a0de-3782ca439256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.759879 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.759686 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ltq\" (UniqueName: \"kubernetes.io/projected/543ea719-e826-47dd-a0de-3782ca439256-kube-api-access-q6ltq\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.759935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.759890 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543ea719-e826-47dd-a0de-3782ca439256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.860953 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.860916 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543ea719-e826-47dd-a0de-3782ca439256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.861116 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.861003 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543ea719-e826-47dd-a0de-3782ca439256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.861116 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.861032 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ltq\" (UniqueName: \"kubernetes.io/projected/543ea719-e826-47dd-a0de-3782ca439256-kube-api-access-q6ltq\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.861828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.861772 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543ea719-e826-47dd-a0de-3782ca439256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.863597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.863572 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543ea719-e826-47dd-a0de-3782ca439256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.881401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.881376 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ltq\" (UniqueName: \"kubernetes.io/projected/543ea719-e826-47dd-a0de-3782ca439256-kube-api-access-q6ltq\") pod \"maas-api-revoke-sub-2e1f5b5f-qsnkf\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:27.969005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:27.968940 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:28.116070 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:28.116042 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2z245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 30 01:30:28.116245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:28.116089 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podUID="b1988474-6586-4686-b98d-34a146461f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 30 01:30:28.264776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:28.264747 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf"] Oct 30 01:30:28.269302 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:30:28.269276 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543ea719_e826_47dd_a0de_3782ca439256.slice/crio-0ca9073de9e1fcc0b9849de33fe55ad2bf0fa16ca50bc74df903a595ff1548ba WatchSource:0}: Error finding container 0ca9073de9e1fcc0b9849de33fe55ad2bf0fa16ca50bc74df903a595ff1548ba: Status 404 returned error can't find the container with id 0ca9073de9e1fcc0b9849de33fe55ad2bf0fa16ca50bc74df903a595ff1548ba Oct 30 01:30:28.279035 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:28.279002 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" event={"ID":"543ea719-e826-47dd-a0de-3782ca439256","Type":"ContainerStarted","Data":"0ca9073de9e1fcc0b9849de33fe55ad2bf0fa16ca50bc74df903a595ff1548ba"} Oct 30 01:30:29.116221 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:29.116191 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2z245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 30 01:30:29.116638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:29.116238 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" podUID="b1988474-6586-4686-b98d-34a146461f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 30 01:30:29.287587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:29.287509 4905 generic.go:356] "Generic (PLEG): container finished" podID="543ea719-e826-47dd-a0de-3782ca439256" containerID="2df362b7828a9da9a5b4cf7d185c9ddd60739b822ff55fa4483ca97b432ccc71" exitCode=0 Oct 30 01:30:29.287794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:29.287624 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" event={"ID":"543ea719-e826-47dd-a0de-3782ca439256","Type":"ContainerDied","Data":"2df362b7828a9da9a5b4cf7d185c9ddd60739b822ff55fa4483ca97b432ccc71"} Oct 30 01:30:30.119955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.119929 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:30.120436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.120332 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:30.121123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.121100 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2z245" Oct 30 01:30:30.443414 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.443385 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:30.487986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.487959 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543ea719-e826-47dd-a0de-3782ca439256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543ea719-e826-47dd-a0de-3782ca439256-maas-api-openshift-ca\") pod \"543ea719-e826-47dd-a0de-3782ca439256\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " Oct 30 01:30:30.488149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.488028 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543ea719-e826-47dd-a0de-3782ca439256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543ea719-e826-47dd-a0de-3782ca439256-maas-api-xks-ca\") pod \"543ea719-e826-47dd-a0de-3782ca439256\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " Oct 30 01:30:30.488149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.488123 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543ea719-e826-47dd-a0de-3782ca439256-kube-api-access-q6ltq\" (UniqueName: \"kubernetes.io/projected/543ea719-e826-47dd-a0de-3782ca439256-kube-api-access-q6ltq\") pod \"543ea719-e826-47dd-a0de-3782ca439256\" (UID: \"543ea719-e826-47dd-a0de-3782ca439256\") " Oct 30 01:30:30.488345 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.488323 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543ea719-e826-47dd-a0de-3782ca439256-maas-api-openshift-ca" pod "543ea719-e826-47dd-a0de-3782ca439256" (UID: "543ea719-e826-47dd-a0de-3782ca439256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:30.490307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.490279 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543ea719-e826-47dd-a0de-3782ca439256-maas-api-xks-ca" pod "543ea719-e826-47dd-a0de-3782ca439256" (UID: "543ea719-e826-47dd-a0de-3782ca439256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:30.490753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.490730 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543ea719-e826-47dd-a0de-3782ca439256-kube-api-access-q6ltq" pod "543ea719-e826-47dd-a0de-3782ca439256" (UID: "543ea719-e826-47dd-a0de-3782ca439256"). InnerVolumeSpecName "kube-api-access-q6ltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:30.589354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.589321 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ltq\" (UniqueName: \"kubernetes.io/projected/543ea719-e826-47dd-a0de-3782ca439256-kube-api-access-q6ltq\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:30.589354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.589350 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543ea719-e826-47dd-a0de-3782ca439256-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:30.589354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:30.589361 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543ea719-e826-47dd-a0de-3782ca439256-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:31.300960 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:31.300929 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" Oct 30 01:30:31.301406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:31.300977 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf" event={"ID":"543ea719-e826-47dd-a0de-3782ca439256","Type":"ContainerDied","Data":"0ca9073de9e1fcc0b9849de33fe55ad2bf0fa16ca50bc74df903a595ff1548ba"} Oct 30 01:30:31.301406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:31.301021 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca9073de9e1fcc0b9849de33fe55ad2bf0fa16ca50bc74df903a595ff1548ba" Oct 30 01:30:33.029996 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:33.029958 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8"] Oct 30 01:30:33.043801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:33.043769 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145baaf-qggg8"] Oct 30 01:30:33.189694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:33.189663 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc" path="/var/lib/kubelet/pods/8e5efe4a-b47a-4f1a-90c4-3484ddddb7bc/volumes" Oct 30 01:30:38.740794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.740759 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz"] Oct 30 01:30:38.741357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.741222 4905 cpu_manager.go:414] "containerMap: removing container" podUID="543ea719-e826-47dd-a0de-3782ca439256" containerName="revoke-keys" Oct 30 01:30:38.741357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.741235 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543ea719-e826-47dd-a0de-3782ca439256" containerName="revoke-keys" Oct 30 01:30:38.741357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.741317 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="543ea719-e826-47dd-a0de-3782ca439256" containerName="revoke-keys" Oct 30 01:30:38.741357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.741328 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7d550f-91f6-4fc6-bb03-1bfe4eeb275b" containerName="cleanup" Oct 30 01:30:38.744559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.744536 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.764589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.764563 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz"] Oct 30 01:30:38.773364 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.773334 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.773513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.773385 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.773513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.773493 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745zl\" (UniqueName: \"kubernetes.io/projected/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-kube-api-access-745zl\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.775335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.775312 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:38.775335 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.775321 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:38.775547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.775365 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:38.775613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.775548 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:30:38.874960 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.874926 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.875134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.874981 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.875134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.875010 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745zl\" (UniqueName: \"kubernetes.io/projected/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-kube-api-access-745zl\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.875720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.875694 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.877502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.877481 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:38.897714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:38.897692 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745zl\" (UniqueName: \"kubernetes.io/projected/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-kube-api-access-745zl\") pod \"maas-api-revoke-sub-ec843018-9kcwz\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:39.059698 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:39.059606 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:39.363211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:39.363177 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz"] Oct 30 01:30:39.363470 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:30:39.363444 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d68dc9_d11a_425e_9a8d_3fca41ed8bb1.slice/crio-8a4983fa6a52eccae8f6187a4c4161600495e5331d55438f7226e5cb45afd522 WatchSource:0}: Error finding container 8a4983fa6a52eccae8f6187a4c4161600495e5331d55438f7226e5cb45afd522: Status 404 returned error can't find the container with id 8a4983fa6a52eccae8f6187a4c4161600495e5331d55438f7226e5cb45afd522 Oct 30 01:30:40.355219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:40.355181 4905 generic.go:356] "Generic (PLEG): container finished" podID="43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" containerID="f5c6bbfa08ed34ac8d8964af755c6e01d0378455645f44604c6a777ea4d2b496" exitCode=0 Oct 30 01:30:40.355664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:40.355228 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" event={"ID":"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1","Type":"ContainerDied","Data":"f5c6bbfa08ed34ac8d8964af755c6e01d0378455645f44604c6a777ea4d2b496"} Oct 30 01:30:40.355664 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:40.355289 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" event={"ID":"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1","Type":"ContainerStarted","Data":"8a4983fa6a52eccae8f6187a4c4161600495e5331d55438f7226e5cb45afd522"} Oct 30 01:30:41.509120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.509092 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:41.603080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.603041 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-xks-ca\") pod \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " Oct 30 01:30:41.603301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.603195 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-openshift-ca\") pod \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " Oct 30 01:30:41.603301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.603240 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-kube-api-access-745zl\" (UniqueName: \"kubernetes.io/projected/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-kube-api-access-745zl\") pod \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\" (UID: \"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1\") " Oct 30 01:30:41.604229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.604177 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-openshift-ca" pod "43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" (UID: "43d68dc9-d11a-425e-9a8d-3fca41ed8bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:41.610580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.606092 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-xks-ca" pod "43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" (UID: "43d68dc9-d11a-425e-9a8d-3fca41ed8bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:41.610580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.606351 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-kube-api-access-745zl" pod "43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" (UID: "43d68dc9-d11a-425e-9a8d-3fca41ed8bb1"). InnerVolumeSpecName "kube-api-access-745zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:41.704331 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.704287 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:41.704331 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.704331 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745zl\" (UniqueName: \"kubernetes.io/projected/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-kube-api-access-745zl\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:41.704633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:41.704348 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.372824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:42.372797 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" Oct 30 01:30:42.373189 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:42.372830 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz" event={"ID":"43d68dc9-d11a-425e-9a8d-3fca41ed8bb1","Type":"ContainerDied","Data":"8a4983fa6a52eccae8f6187a4c4161600495e5331d55438f7226e5cb45afd522"} Oct 30 01:30:42.373299 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:42.373223 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a4983fa6a52eccae8f6187a4c4161600495e5331d55438f7226e5cb45afd522" Oct 30 01:30:42.425644 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:30:42.425591 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:30:49.306563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:30:49.306497 4905 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-171.ec2.internal" podUID="92d13e825dfc8d10deb63823b2be0da4" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 01:31:06.387756 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.387718 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7"] Oct 30 01:31:06.388531 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.388494 4905 cpu_manager.go:414] "containerMap: removing container" podUID="43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" containerName="revoke-keys" Oct 30 01:31:06.388636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.388536 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" containerName="revoke-keys" Oct 30 01:31:06.388677 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.388651 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" containerName="revoke-keys" Oct 30 01:31:06.391818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.391801 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.413462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.413428 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7"] Oct 30 01:31:06.422838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.422805 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:31:06.422976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.422880 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:31:06.422976 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.422901 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:31:06.423230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.423211 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:31:06.541021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.540989 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmfg\" (UniqueName: \"kubernetes.io/projected/3a634fd1-d117-4287-a406-44db7630535f-kube-api-access-jtmfg\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.541170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.541043 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a634fd1-d117-4287-a406-44db7630535f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.541170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.541136 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a634fd1-d117-4287-a406-44db7630535f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.642573 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.642495 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a634fd1-d117-4287-a406-44db7630535f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.642690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.642583 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmfg\" (UniqueName: \"kubernetes.io/projected/3a634fd1-d117-4287-a406-44db7630535f-kube-api-access-jtmfg\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.642690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.642619 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a634fd1-d117-4287-a406-44db7630535f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.643137 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.643116 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a634fd1-d117-4287-a406-44db7630535f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.645010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.644991 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a634fd1-d117-4287-a406-44db7630535f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.663371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.663348 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmfg\" (UniqueName: \"kubernetes.io/projected/3a634fd1-d117-4287-a406-44db7630535f-kube-api-access-jtmfg\") pod \"maas-api-revoke-sub-046e1343-7kpv7\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:06.705342 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:06.705323 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:07.003350 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:31:07.003322 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a634fd1_d117_4287_a406_44db7630535f.slice/crio-c2a5474fc3814581a42a5bc76d7e8180fbcd1325374d2286d3aed11a7de05ec8 WatchSource:0}: Error finding container c2a5474fc3814581a42a5bc76d7e8180fbcd1325374d2286d3aed11a7de05ec8: Status 404 returned error can't find the container with id c2a5474fc3814581a42a5bc76d7e8180fbcd1325374d2286d3aed11a7de05ec8 Oct 30 01:31:07.003476 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:07.003454 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7"] Oct 30 01:31:07.560556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:07.560494 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" event={"ID":"3a634fd1-d117-4287-a406-44db7630535f","Type":"ContainerStarted","Data":"d13b6d6e4260de656a32dad4676b2eb3737fd41e59be96c0a7f4ca3e725850d4"} Oct 30 01:31:07.561010 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:07.560585 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" event={"ID":"3a634fd1-d117-4287-a406-44db7630535f","Type":"ContainerStarted","Data":"c2a5474fc3814581a42a5bc76d7e8180fbcd1325374d2286d3aed11a7de05ec8"} Oct 30 01:31:07.602790 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:07.602696 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" podStartSLOduration=1.602682974 podStartE2EDuration="1.602682974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:07.601206151 +0000 UTC m=+3152.544379344" watchObservedRunningTime="2026-10-30 01:31:07.602682974 +0000 UTC m=+3152.545856167" Oct 30 01:31:08.566972 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:08.566933 4905 generic.go:356] "Generic (PLEG): container finished" podID="3a634fd1-d117-4287-a406-44db7630535f" containerID="d13b6d6e4260de656a32dad4676b2eb3737fd41e59be96c0a7f4ca3e725850d4" exitCode=0 Oct 30 01:31:08.567442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:08.567010 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" event={"ID":"3a634fd1-d117-4287-a406-44db7630535f","Type":"ContainerDied","Data":"d13b6d6e4260de656a32dad4676b2eb3737fd41e59be96c0a7f4ca3e725850d4"} Oct 30 01:31:09.714762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.714735 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:09.872583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.872480 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a634fd1-d117-4287-a406-44db7630535f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a634fd1-d117-4287-a406-44db7630535f-maas-api-openshift-ca\") pod \"3a634fd1-d117-4287-a406-44db7630535f\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " Oct 30 01:31:09.872583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.872551 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a634fd1-d117-4287-a406-44db7630535f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a634fd1-d117-4287-a406-44db7630535f-maas-api-xks-ca\") pod \"3a634fd1-d117-4287-a406-44db7630535f\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " Oct 30 01:31:09.872795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.872640 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a634fd1-d117-4287-a406-44db7630535f-kube-api-access-jtmfg\" (UniqueName: \"kubernetes.io/projected/3a634fd1-d117-4287-a406-44db7630535f-kube-api-access-jtmfg\") pod \"3a634fd1-d117-4287-a406-44db7630535f\" (UID: \"3a634fd1-d117-4287-a406-44db7630535f\") " Oct 30 01:31:09.872896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.872876 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a634fd1-d117-4287-a406-44db7630535f-maas-api-openshift-ca" pod "3a634fd1-d117-4287-a406-44db7630535f" (UID: "3a634fd1-d117-4287-a406-44db7630535f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:09.873038 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.873022 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a634fd1-d117-4287-a406-44db7630535f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:09.874785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.874760 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a634fd1-d117-4287-a406-44db7630535f-maas-api-xks-ca" pod "3a634fd1-d117-4287-a406-44db7630535f" (UID: "3a634fd1-d117-4287-a406-44db7630535f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:09.874918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.874841 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a634fd1-d117-4287-a406-44db7630535f-kube-api-access-jtmfg" pod "3a634fd1-d117-4287-a406-44db7630535f" (UID: "3a634fd1-d117-4287-a406-44db7630535f"). InnerVolumeSpecName "kube-api-access-jtmfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:09.974187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.974160 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a634fd1-d117-4287-a406-44db7630535f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:09.974187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:09.974184 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtmfg\" (UniqueName: \"kubernetes.io/projected/3a634fd1-d117-4287-a406-44db7630535f-kube-api-access-jtmfg\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:10.587413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:10.587369 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" event={"ID":"3a634fd1-d117-4287-a406-44db7630535f","Type":"ContainerDied","Data":"c2a5474fc3814581a42a5bc76d7e8180fbcd1325374d2286d3aed11a7de05ec8"} Oct 30 01:31:10.587413 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:10.587399 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7" Oct 30 01:31:10.587669 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:10.587458 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2a5474fc3814581a42a5bc76d7e8180fbcd1325374d2286d3aed11a7de05ec8" Oct 30 01:31:12.413743 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:31:12.413710 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:31:16.684004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.683973 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph"] Oct 30 01:31:16.684658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.684638 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3a634fd1-d117-4287-a406-44db7630535f" containerName="revoke-keys" Oct 30 01:31:16.684789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.684660 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a634fd1-d117-4287-a406-44db7630535f" containerName="revoke-keys" Oct 30 01:31:16.684851 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.684794 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a634fd1-d117-4287-a406-44db7630535f" containerName="revoke-keys" Oct 30 01:31:16.688229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.688208 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.720182 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.720154 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:31:16.720391 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.720368 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:31:16.720503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.720488 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:31:16.720613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.720599 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:31:16.722846 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.722818 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph"] Oct 30 01:31:16.838135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.838105 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.838343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.838141 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvdk\" (UniqueName: \"kubernetes.io/projected/f6c65ce8-b406-489d-b51a-87c36444cbe5-kube-api-access-4pvdk\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.838343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.838321 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.939682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.939597 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.939682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.939631 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvdk\" (UniqueName: \"kubernetes.io/projected/f6c65ce8-b406-489d-b51a-87c36444cbe5-kube-api-access-4pvdk\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.939904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.939687 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.940303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.940274 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.942113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.942092 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:16.961491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:16.961456 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvdk\" (UniqueName: \"kubernetes.io/projected/f6c65ce8-b406-489d-b51a-87c36444cbe5-kube-api-access-4pvdk\") pod \"maas-api-revoke-sub-d3fba056-bf5ph\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:17.000841 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:17.000821 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:17.298555 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:31:17.298507 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c65ce8_b406_489d_b51a_87c36444cbe5.slice/crio-29e19f90ffa1c1b8b4520dc9e00e104323937702c9bbc541b35c8ebaf2edaf69 WatchSource:0}: Error finding container 29e19f90ffa1c1b8b4520dc9e00e104323937702c9bbc541b35c8ebaf2edaf69: Status 404 returned error can't find the container with id 29e19f90ffa1c1b8b4520dc9e00e104323937702c9bbc541b35c8ebaf2edaf69 Oct 30 01:31:17.299460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:17.299428 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph"] Oct 30 01:31:17.624848 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:17.624759 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" event={"ID":"f6c65ce8-b406-489d-b51a-87c36444cbe5","Type":"ContainerStarted","Data":"29e19f90ffa1c1b8b4520dc9e00e104323937702c9bbc541b35c8ebaf2edaf69"} Oct 30 01:31:18.635151 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:18.635112 4905 generic.go:356] "Generic (PLEG): container finished" podID="f6c65ce8-b406-489d-b51a-87c36444cbe5" containerID="e270addc52cab35c394401c3675b9aeeb6019ecd926779e46dcd1d0dd2342c1a" exitCode=0 Oct 30 01:31:18.635712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:18.635166 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" event={"ID":"f6c65ce8-b406-489d-b51a-87c36444cbe5","Type":"ContainerDied","Data":"e270addc52cab35c394401c3675b9aeeb6019ecd926779e46dcd1d0dd2342c1a"} Oct 30 01:31:20.575545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.574708 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:20.704227 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.704188 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" event={"ID":"f6c65ce8-b406-489d-b51a-87c36444cbe5","Type":"ContainerDied","Data":"29e19f90ffa1c1b8b4520dc9e00e104323937702c9bbc541b35c8ebaf2edaf69"} Oct 30 01:31:20.726805 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.726779 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e19f90ffa1c1b8b4520dc9e00e104323937702c9bbc541b35c8ebaf2edaf69" Oct 30 01:31:20.727135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.727118 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph" Oct 30 01:31:20.751966 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.751938 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c65ce8-b406-489d-b51a-87c36444cbe5-kube-api-access-4pvdk\" (UniqueName: \"kubernetes.io/projected/f6c65ce8-b406-489d-b51a-87c36444cbe5-kube-api-access-4pvdk\") pod \"f6c65ce8-b406-489d-b51a-87c36444cbe5\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " Oct 30 01:31:20.752211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.752190 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-openshift-ca\") pod \"f6c65ce8-b406-489d-b51a-87c36444cbe5\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " Oct 30 01:31:20.752416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.752397 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-xks-ca\") pod \"f6c65ce8-b406-489d-b51a-87c36444cbe5\" (UID: \"f6c65ce8-b406-489d-b51a-87c36444cbe5\") " Oct 30 01:31:20.757205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.757166 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-openshift-ca" pod "f6c65ce8-b406-489d-b51a-87c36444cbe5" (UID: "f6c65ce8-b406-489d-b51a-87c36444cbe5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:20.816268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.816236 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-xks-ca" pod "f6c65ce8-b406-489d-b51a-87c36444cbe5" (UID: "f6c65ce8-b406-489d-b51a-87c36444cbe5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:20.816622 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.816599 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c65ce8-b406-489d-b51a-87c36444cbe5-kube-api-access-4pvdk" pod "f6c65ce8-b406-489d-b51a-87c36444cbe5" (UID: "f6c65ce8-b406-489d-b51a-87c36444cbe5"). InnerVolumeSpecName "kube-api-access-4pvdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:20.856395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.855999 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pvdk\" (UniqueName: \"kubernetes.io/projected/f6c65ce8-b406-489d-b51a-87c36444cbe5-kube-api-access-4pvdk\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:20.856395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.856039 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:20.856395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:20.856054 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c65ce8-b406-489d-b51a-87c36444cbe5-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:38.123193 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.123162 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb"] Oct 30 01:31:38.123738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.123633 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f6c65ce8-b406-489d-b51a-87c36444cbe5" containerName="revoke-keys" Oct 30 01:31:38.123738 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.123649 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c65ce8-b406-489d-b51a-87c36444cbe5" containerName="revoke-keys" Oct 30 01:31:38.123926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.123789 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c65ce8-b406-489d-b51a-87c36444cbe5" containerName="revoke-keys" Oct 30 01:31:38.127808 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.127785 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.144711 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.144672 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:31:38.158578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.158553 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:31:38.158730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.158593 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:31:38.158730 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.158650 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:31:38.161329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.161301 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb"] Oct 30 01:31:38.268279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.268247 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.268428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.268299 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8v2\" (UniqueName: \"kubernetes.io/projected/46d632d0-49d2-4273-93ed-b74df007430e-kube-api-access-7x8v2\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.268428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.268402 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.369146 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.369118 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.369277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.369189 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.369277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.369219 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8v2\" (UniqueName: \"kubernetes.io/projected/46d632d0-49d2-4273-93ed-b74df007430e-kube-api-access-7x8v2\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.369898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.369878 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.371620 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.371602 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.390588 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.390503 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8v2\" (UniqueName: \"kubernetes.io/projected/46d632d0-49d2-4273-93ed-b74df007430e-kube-api-access-7x8v2\") pod \"maas-api-revoke-sub-ae594979-xnltb\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.441401 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.441379 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:38.736966 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:31:38.736937 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d632d0_49d2_4273_93ed_b74df007430e.slice/crio-dc52c7dee5cd2eb0ec166d1c63d6a765ac2c1251966df9f0e395ea6abf35abe4 WatchSource:0}: Error finding container dc52c7dee5cd2eb0ec166d1c63d6a765ac2c1251966df9f0e395ea6abf35abe4: Status 404 returned error can't find the container with id dc52c7dee5cd2eb0ec166d1c63d6a765ac2c1251966df9f0e395ea6abf35abe4 Oct 30 01:31:38.737656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.737630 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb"] Oct 30 01:31:38.824691 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:38.824665 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" event={"ID":"46d632d0-49d2-4273-93ed-b74df007430e","Type":"ContainerStarted","Data":"dc52c7dee5cd2eb0ec166d1c63d6a765ac2c1251966df9f0e395ea6abf35abe4"} Oct 30 01:31:39.830911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:39.830843 4905 generic.go:356] "Generic (PLEG): container finished" podID="46d632d0-49d2-4273-93ed-b74df007430e" containerID="cf6233c3f97f87422ed90905620bd7be60400f1b1113a5e99c74fc98c9fa1db9" exitCode=0 Oct 30 01:31:39.830911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:39.830884 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" event={"ID":"46d632d0-49d2-4273-93ed-b74df007430e","Type":"ContainerDied","Data":"cf6233c3f97f87422ed90905620bd7be60400f1b1113a5e99c74fc98c9fa1db9"} Oct 30 01:31:40.972152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:40.972128 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:41.092640 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.092575 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-xks-ca\") pod \"46d632d0-49d2-4273-93ed-b74df007430e\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " Oct 30 01:31:41.092770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.092687 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d632d0-49d2-4273-93ed-b74df007430e-kube-api-access-7x8v2\" (UniqueName: \"kubernetes.io/projected/46d632d0-49d2-4273-93ed-b74df007430e-kube-api-access-7x8v2\") pod \"46d632d0-49d2-4273-93ed-b74df007430e\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " Oct 30 01:31:41.092770 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.092730 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-openshift-ca\") pod \"46d632d0-49d2-4273-93ed-b74df007430e\" (UID: \"46d632d0-49d2-4273-93ed-b74df007430e\") " Oct 30 01:31:41.093095 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.093068 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-openshift-ca" pod "46d632d0-49d2-4273-93ed-b74df007430e" (UID: "46d632d0-49d2-4273-93ed-b74df007430e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:41.094720 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.094696 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-xks-ca" pod "46d632d0-49d2-4273-93ed-b74df007430e" (UID: "46d632d0-49d2-4273-93ed-b74df007430e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:41.094937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.094917 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d632d0-49d2-4273-93ed-b74df007430e-kube-api-access-7x8v2" pod "46d632d0-49d2-4273-93ed-b74df007430e" (UID: "46d632d0-49d2-4273-93ed-b74df007430e"). InnerVolumeSpecName "kube-api-access-7x8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:41.193974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.193947 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x8v2\" (UniqueName: \"kubernetes.io/projected/46d632d0-49d2-4273-93ed-b74df007430e-kube-api-access-7x8v2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:41.193974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.193974 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:41.194148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.193989 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d632d0-49d2-4273-93ed-b74df007430e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:31:41.843240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.843210 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" Oct 30 01:31:41.843240 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.843224 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb" event={"ID":"46d632d0-49d2-4273-93ed-b74df007430e","Type":"ContainerDied","Data":"dc52c7dee5cd2eb0ec166d1c63d6a765ac2c1251966df9f0e395ea6abf35abe4"} Oct 30 01:31:41.843614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:41.843272 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc52c7dee5cd2eb0ec166d1c63d6a765ac2c1251966df9f0e395ea6abf35abe4" Oct 30 01:31:42.414189 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:31:42.414155 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:31:47.118054 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:47.117981 4905 scope.go:122] "RemoveContainer" containerID="1c10b19c7754c9b43c9c2af6e01cad5ea51de7c194279456291810bfe8d46cbf" Oct 30 01:31:59.513983 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.513947 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj"] Oct 30 01:31:59.514637 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.514618 4905 cpu_manager.go:414] "containerMap: removing container" podUID="46d632d0-49d2-4273-93ed-b74df007430e" containerName="revoke-keys" Oct 30 01:31:59.514706 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.514641 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d632d0-49d2-4273-93ed-b74df007430e" containerName="revoke-keys" Oct 30 01:31:59.514803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.514789 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d632d0-49d2-4273-93ed-b74df007430e" containerName="revoke-keys" Oct 30 01:31:59.519437 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.519413 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.526649 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.526628 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.526757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.526658 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnr85\" (UniqueName: \"kubernetes.io/projected/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-kube-api-access-gnr85\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.526937 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.526906 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.538483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.538456 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj"] Oct 30 01:31:59.549136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.549102 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:31:59.549260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.549182 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:31:59.549260 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.549222 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:31:59.549871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.549846 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:31:59.627859 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.627832 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.628014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.627895 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.628014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.627921 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnr85\" (UniqueName: \"kubernetes.io/projected/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-kube-api-access-gnr85\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.628599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.628579 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.630273 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.630255 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.648652 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.648633 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnr85\" (UniqueName: \"kubernetes.io/projected/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-kube-api-access-gnr85\") pod \"maas-api-revoke-sub-a4aef7d7-2lxpj\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:31:59.832889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:31:59.832833 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:32:00.137925 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:32:00.137895 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89bc660_a9ed_4e1c_81fe_9704b81a0dd4.slice/crio-5d4ae509c77179b6e85e524494ad7b72fc198cfda6e09815e5e4c1ccf856ec01 WatchSource:0}: Error finding container 5d4ae509c77179b6e85e524494ad7b72fc198cfda6e09815e5e4c1ccf856ec01: Status 404 returned error can't find the container with id 5d4ae509c77179b6e85e524494ad7b72fc198cfda6e09815e5e4c1ccf856ec01 Oct 30 01:32:00.138049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:00.137988 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj"] Oct 30 01:32:00.968638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:00.968608 4905 generic.go:356] "Generic (PLEG): container finished" podID="e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" containerID="e850713337e5ae2b8c436c31ea2853cffa1537680568f430b7fa2aac4c854f31" exitCode=0 Oct 30 01:32:00.969008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:00.968697 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" event={"ID":"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4","Type":"ContainerDied","Data":"e850713337e5ae2b8c436c31ea2853cffa1537680568f430b7fa2aac4c854f31"} Oct 30 01:32:00.969008 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:00.968771 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" event={"ID":"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4","Type":"ContainerStarted","Data":"5d4ae509c77179b6e85e524494ad7b72fc198cfda6e09815e5e4c1ccf856ec01"} Oct 30 01:32:02.104416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.104392 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:32:02.147772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.147741 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-openshift-ca\") pod \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " Oct 30 01:32:02.147911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.147817 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-kube-api-access-gnr85\" (UniqueName: \"kubernetes.io/projected/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-kube-api-access-gnr85\") pod \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " Oct 30 01:32:02.147911 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.147873 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-xks-ca\") pod \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\" (UID: \"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4\") " Oct 30 01:32:02.148133 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.148111 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-openshift-ca" pod "e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" (UID: "e89bc660-a9ed-4e1c-81fe-9704b81a0dd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:02.150675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.150643 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-xks-ca" pod "e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" (UID: "e89bc660-a9ed-4e1c-81fe-9704b81a0dd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:02.152732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.152705 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-kube-api-access-gnr85" pod "e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" (UID: "e89bc660-a9ed-4e1c-81fe-9704b81a0dd4"). InnerVolumeSpecName "kube-api-access-gnr85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:02.249789 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.249758 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:02.249944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.249793 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnr85\" (UniqueName: \"kubernetes.io/projected/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-kube-api-access-gnr85\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:02.249944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.249814 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:02.980693 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.980666 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" Oct 30 01:32:02.980874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.980694 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj" event={"ID":"e89bc660-a9ed-4e1c-81fe-9704b81a0dd4","Type":"ContainerDied","Data":"5d4ae509c77179b6e85e524494ad7b72fc198cfda6e09815e5e4c1ccf856ec01"} Oct 30 01:32:02.980874 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:02.980734 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d4ae509c77179b6e85e524494ad7b72fc198cfda6e09815e5e4c1ccf856ec01" Oct 30 01:32:12.413677 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:32:12.413643 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:32:19.430441 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.430403 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm"] Oct 30 01:32:19.430865 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.430858 4905 cpu_manager.go:414] "containerMap: removing container" podUID="e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" containerName="revoke-keys" Oct 30 01:32:19.430915 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.430869 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" containerName="revoke-keys" Oct 30 01:32:19.430955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.430939 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" containerName="revoke-keys" Oct 30 01:32:19.434242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.434227 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.454882 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.454852 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm"] Oct 30 01:32:19.465449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.465426 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:19.465632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.465451 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:19.465632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.465450 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:19.465632 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.465606 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:32:19.527853 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.527829 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkj4\" (UniqueName: \"kubernetes.io/projected/623af2c3-1233-46fb-9e46-76acaf0c3104-kube-api-access-wgkj4\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.527979 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.527903 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.528021 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.528001 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.629371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.629342 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkj4\" (UniqueName: \"kubernetes.io/projected/623af2c3-1233-46fb-9e46-76acaf0c3104-kube-api-access-wgkj4\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.629492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.629389 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.629492 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.629425 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.630038 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.630014 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.631899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.631874 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.650558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.650537 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkj4\" (UniqueName: \"kubernetes.io/projected/623af2c3-1233-46fb-9e46-76acaf0c3104-kube-api-access-wgkj4\") pod \"maas-api-revoke-sub-79805ca9-k5qqm\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:19.758800 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:19.758767 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:20.052426 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:32:20.052395 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623af2c3_1233_46fb_9e46_76acaf0c3104.slice/crio-7cc96f3f9e931c2268940186e86ca5c43620b4a653c8ec3b6201b42b79b40456 WatchSource:0}: Error finding container 7cc96f3f9e931c2268940186e86ca5c43620b4a653c8ec3b6201b42b79b40456: Status 404 returned error can't find the container with id 7cc96f3f9e931c2268940186e86ca5c43620b4a653c8ec3b6201b42b79b40456 Oct 30 01:32:20.053795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:20.053764 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm"] Oct 30 01:32:20.156205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:20.156173 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" event={"ID":"623af2c3-1233-46fb-9e46-76acaf0c3104","Type":"ContainerStarted","Data":"7cc96f3f9e931c2268940186e86ca5c43620b4a653c8ec3b6201b42b79b40456"} Oct 30 01:32:21.162930 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:21.162855 4905 generic.go:356] "Generic (PLEG): container finished" podID="623af2c3-1233-46fb-9e46-76acaf0c3104" containerID="2af246774e4181028ec56f049d8aa40980371be4f7a1348014066b648fe64715" exitCode=0 Oct 30 01:32:21.163302 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:21.162951 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" event={"ID":"623af2c3-1233-46fb-9e46-76acaf0c3104","Type":"ContainerDied","Data":"2af246774e4181028ec56f049d8aa40980371be4f7a1348014066b648fe64715"} Oct 30 01:32:22.302175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.302152 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:22.351910 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.351877 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623af2c3-1233-46fb-9e46-76acaf0c3104-kube-api-access-wgkj4\" (UniqueName: \"kubernetes.io/projected/623af2c3-1233-46fb-9e46-76acaf0c3104-kube-api-access-wgkj4\") pod \"623af2c3-1233-46fb-9e46-76acaf0c3104\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " Oct 30 01:32:22.352047 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.351986 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-xks-ca\") pod \"623af2c3-1233-46fb-9e46-76acaf0c3104\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " Oct 30 01:32:22.352187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.352048 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-openshift-ca\") pod \"623af2c3-1233-46fb-9e46-76acaf0c3104\" (UID: \"623af2c3-1233-46fb-9e46-76acaf0c3104\") " Oct 30 01:32:22.352450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.352425 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-openshift-ca" pod "623af2c3-1233-46fb-9e46-76acaf0c3104" (UID: "623af2c3-1233-46fb-9e46-76acaf0c3104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:22.354293 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.354270 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-xks-ca" pod "623af2c3-1233-46fb-9e46-76acaf0c3104" (UID: "623af2c3-1233-46fb-9e46-76acaf0c3104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:22.354639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.354623 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623af2c3-1233-46fb-9e46-76acaf0c3104-kube-api-access-wgkj4" pod "623af2c3-1233-46fb-9e46-76acaf0c3104" (UID: "623af2c3-1233-46fb-9e46-76acaf0c3104"). InnerVolumeSpecName "kube-api-access-wgkj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:22.453194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.453143 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgkj4\" (UniqueName: \"kubernetes.io/projected/623af2c3-1233-46fb-9e46-76acaf0c3104-kube-api-access-wgkj4\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:22.453194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.453167 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:22.453194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:22.453177 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623af2c3-1233-46fb-9e46-76acaf0c3104-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:23.174105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:23.174079 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" Oct 30 01:32:23.174282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:23.174108 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm" event={"ID":"623af2c3-1233-46fb-9e46-76acaf0c3104","Type":"ContainerDied","Data":"7cc96f3f9e931c2268940186e86ca5c43620b4a653c8ec3b6201b42b79b40456"} Oct 30 01:32:23.174282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:23.174150 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cc96f3f9e931c2268940186e86ca5c43620b4a653c8ec3b6201b42b79b40456" Oct 30 01:32:37.247665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.247628 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk"] Oct 30 01:32:37.248175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.248086 4905 cpu_manager.go:414] "containerMap: removing container" podUID="623af2c3-1233-46fb-9e46-76acaf0c3104" containerName="revoke-keys" Oct 30 01:32:37.248175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.248099 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623af2c3-1233-46fb-9e46-76acaf0c3104" containerName="revoke-keys" Oct 30 01:32:37.248307 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.248213 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="623af2c3-1233-46fb-9e46-76acaf0c3104" containerName="revoke-keys" Oct 30 01:32:37.251316 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.251292 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.286939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.286902 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk"] Oct 30 01:32:37.287774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.287745 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:37.287968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.287848 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:32:37.288140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.288003 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:37.288140 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.288040 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:37.342351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.342322 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5992f34e-bc76-4641-973f-7461355cd585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.342564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.342548 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5992f34e-bc76-4641-973f-7461355cd585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.342629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.342583 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqgz\" (UniqueName: \"kubernetes.io/projected/5992f34e-bc76-4641-973f-7461355cd585-kube-api-access-jkqgz\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.443376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.443345 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5992f34e-bc76-4641-973f-7461355cd585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.443560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.443383 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqgz\" (UniqueName: \"kubernetes.io/projected/5992f34e-bc76-4641-973f-7461355cd585-kube-api-access-jkqgz\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.443560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.443401 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5992f34e-bc76-4641-973f-7461355cd585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.444081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.444062 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5992f34e-bc76-4641-973f-7461355cd585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.445747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.445725 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5992f34e-bc76-4641-973f-7461355cd585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.465172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.465149 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqgz\" (UniqueName: \"kubernetes.io/projected/5992f34e-bc76-4641-973f-7461355cd585-kube-api-access-jkqgz\") pod \"maas-api-revoke-sub-0b7e8edb-d5pxk\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.569077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.569023 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:37.865100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:37.865076 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk"] Oct 30 01:32:37.867005 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:32:37.866976 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5992f34e_bc76_4641_973f_7461355cd585.slice/crio-1ac909023d77720937f3fc5e08f6a9605d13fc5b4d605fd34e63f124c90424d8 WatchSource:0}: Error finding container 1ac909023d77720937f3fc5e08f6a9605d13fc5b4d605fd34e63f124c90424d8: Status 404 returned error can't find the container with id 1ac909023d77720937f3fc5e08f6a9605d13fc5b4d605fd34e63f124c90424d8 Oct 30 01:32:38.303512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:38.303477 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" event={"ID":"5992f34e-bc76-4641-973f-7461355cd585","Type":"ContainerStarted","Data":"c4e27045a8932391d2a70f72efe8240a010797da031f43719eb7311cd1b7b201"} Oct 30 01:32:38.303942 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:38.303550 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" event={"ID":"5992f34e-bc76-4641-973f-7461355cd585","Type":"ContainerStarted","Data":"1ac909023d77720937f3fc5e08f6a9605d13fc5b4d605fd34e63f124c90424d8"} Oct 30 01:32:38.350574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:38.350512 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" podStartSLOduration=1.350498834 podStartE2EDuration="1.350498834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:38.349753562 +0000 UTC m=+3243.292926754" watchObservedRunningTime="2026-10-30 01:32:38.350498834 +0000 UTC m=+3243.293672026" Oct 30 01:32:39.309638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:39.309604 4905 generic.go:356] "Generic (PLEG): container finished" podID="5992f34e-bc76-4641-973f-7461355cd585" containerID="c4e27045a8932391d2a70f72efe8240a010797da031f43719eb7311cd1b7b201" exitCode=0 Oct 30 01:32:39.310085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:39.309646 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" event={"ID":"5992f34e-bc76-4641-973f-7461355cd585","Type":"ContainerDied","Data":"c4e27045a8932391d2a70f72efe8240a010797da031f43719eb7311cd1b7b201"} Oct 30 01:32:40.457439 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.457414 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:40.568998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.568931 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5992f34e-bc76-4641-973f-7461355cd585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5992f34e-bc76-4641-973f-7461355cd585-maas-api-xks-ca\") pod \"5992f34e-bc76-4641-973f-7461355cd585\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " Oct 30 01:32:40.569136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.569016 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5992f34e-bc76-4641-973f-7461355cd585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5992f34e-bc76-4641-973f-7461355cd585-maas-api-openshift-ca\") pod \"5992f34e-bc76-4641-973f-7461355cd585\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " Oct 30 01:32:40.569136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.569064 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5992f34e-bc76-4641-973f-7461355cd585-kube-api-access-jkqgz\" (UniqueName: \"kubernetes.io/projected/5992f34e-bc76-4641-973f-7461355cd585-kube-api-access-jkqgz\") pod \"5992f34e-bc76-4641-973f-7461355cd585\" (UID: \"5992f34e-bc76-4641-973f-7461355cd585\") " Oct 30 01:32:40.569410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.569381 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5992f34e-bc76-4641-973f-7461355cd585-maas-api-openshift-ca" pod "5992f34e-bc76-4641-973f-7461355cd585" (UID: "5992f34e-bc76-4641-973f-7461355cd585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:40.571246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.571218 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5992f34e-bc76-4641-973f-7461355cd585-kube-api-access-jkqgz" pod "5992f34e-bc76-4641-973f-7461355cd585" (UID: "5992f34e-bc76-4641-973f-7461355cd585"). InnerVolumeSpecName "kube-api-access-jkqgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:40.571380 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.571266 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5992f34e-bc76-4641-973f-7461355cd585-maas-api-xks-ca" pod "5992f34e-bc76-4641-973f-7461355cd585" (UID: "5992f34e-bc76-4641-973f-7461355cd585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:40.670083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.670060 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5992f34e-bc76-4641-973f-7461355cd585-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:40.670083 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.670080 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqgz\" (UniqueName: \"kubernetes.io/projected/5992f34e-bc76-4641-973f-7461355cd585-kube-api-access-jkqgz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:40.670239 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:40.670090 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5992f34e-bc76-4641-973f-7461355cd585-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:41.321891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:41.321864 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" Oct 30 01:32:41.322056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:41.321863 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk" event={"ID":"5992f34e-bc76-4641-973f-7461355cd585","Type":"ContainerDied","Data":"1ac909023d77720937f3fc5e08f6a9605d13fc5b4d605fd34e63f124c90424d8"} Oct 30 01:32:41.322056 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:41.321996 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac909023d77720937f3fc5e08f6a9605d13fc5b4d605fd34e63f124c90424d8" Oct 30 01:32:42.414243 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:32:42.414204 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:32:47.965615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.965574 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs"] Oct 30 01:32:47.966467 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.966444 4905 cpu_manager.go:414] "containerMap: removing container" podUID="5992f34e-bc76-4641-973f-7461355cd585" containerName="revoke-keys" Oct 30 01:32:47.966548 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.966471 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5992f34e-bc76-4641-973f-7461355cd585" containerName="revoke-keys" Oct 30 01:32:47.966660 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.966646 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="5992f34e-bc76-4641-973f-7461355cd585" containerName="revoke-keys" Oct 30 01:32:47.970597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.970576 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:47.989301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.989271 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs"] Oct 30 01:32:47.999178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.999154 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:47.999351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:47.999153 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:48.000999 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.000976 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:32:48.001116 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.000996 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:48.041069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.041039 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9d2\" (UniqueName: \"kubernetes.io/projected/f302c26f-43ab-4212-95f0-f00e05a6aca4-kube-api-access-xw9d2\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.041209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.041080 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.041209 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.041111 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.142461 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.142429 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9d2\" (UniqueName: \"kubernetes.io/projected/f302c26f-43ab-4212-95f0-f00e05a6aca4-kube-api-access-xw9d2\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.142656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.142466 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.142656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.142498 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.143120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.143091 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.144890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.144869 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.164023 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.163974 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9d2\" (UniqueName: \"kubernetes.io/projected/f302c26f-43ab-4212-95f0-f00e05a6aca4-kube-api-access-xw9d2\") pod \"maas-api-revoke-sub-3d4156f1-nhbcs\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.285115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.285091 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:48.583723 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:32:48.583676 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf302c26f_43ab_4212_95f0_f00e05a6aca4.slice/crio-9d5924d83de4a1a7aa42c9d5f3274dc03575d23a51418f0c434594ab898b8109 WatchSource:0}: Error finding container 9d5924d83de4a1a7aa42c9d5f3274dc03575d23a51418f0c434594ab898b8109: Status 404 returned error can't find the container with id 9d5924d83de4a1a7aa42c9d5f3274dc03575d23a51418f0c434594ab898b8109 Oct 30 01:32:48.584002 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:48.583981 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs"] Oct 30 01:32:49.371024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:49.370998 4905 generic.go:356] "Generic (PLEG): container finished" podID="f302c26f-43ab-4212-95f0-f00e05a6aca4" containerID="8fd85e5ab5d6e4c0d821fe44a392dce8293702c90216d76cbab80be14b5ebba2" exitCode=0 Oct 30 01:32:49.371355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:49.371070 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" event={"ID":"f302c26f-43ab-4212-95f0-f00e05a6aca4","Type":"ContainerDied","Data":"8fd85e5ab5d6e4c0d821fe44a392dce8293702c90216d76cbab80be14b5ebba2"} Oct 30 01:32:49.371355 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:49.371105 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" event={"ID":"f302c26f-43ab-4212-95f0-f00e05a6aca4","Type":"ContainerStarted","Data":"9d5924d83de4a1a7aa42c9d5f3274dc03575d23a51418f0c434594ab898b8109"} Oct 30 01:32:50.509912 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.509882 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:50.570411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.570375 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-xks-ca\") pod \"f302c26f-43ab-4212-95f0-f00e05a6aca4\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " Oct 30 01:32:50.570590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.570434 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-openshift-ca\") pod \"f302c26f-43ab-4212-95f0-f00e05a6aca4\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " Oct 30 01:32:50.570590 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.570494 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f302c26f-43ab-4212-95f0-f00e05a6aca4-kube-api-access-xw9d2\" (UniqueName: \"kubernetes.io/projected/f302c26f-43ab-4212-95f0-f00e05a6aca4-kube-api-access-xw9d2\") pod \"f302c26f-43ab-4212-95f0-f00e05a6aca4\" (UID: \"f302c26f-43ab-4212-95f0-f00e05a6aca4\") " Oct 30 01:32:50.570837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.570815 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-openshift-ca" pod "f302c26f-43ab-4212-95f0-f00e05a6aca4" (UID: "f302c26f-43ab-4212-95f0-f00e05a6aca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:50.571009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.570992 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:50.573014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.572979 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-xks-ca" pod "f302c26f-43ab-4212-95f0-f00e05a6aca4" (UID: "f302c26f-43ab-4212-95f0-f00e05a6aca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:50.573183 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.573157 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f302c26f-43ab-4212-95f0-f00e05a6aca4-kube-api-access-xw9d2" pod "f302c26f-43ab-4212-95f0-f00e05a6aca4" (UID: "f302c26f-43ab-4212-95f0-f00e05a6aca4"). InnerVolumeSpecName "kube-api-access-xw9d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:50.672196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.672141 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw9d2\" (UniqueName: \"kubernetes.io/projected/f302c26f-43ab-4212-95f0-f00e05a6aca4-kube-api-access-xw9d2\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:50.672196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:50.672168 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f302c26f-43ab-4212-95f0-f00e05a6aca4-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:32:51.384030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:51.384003 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" Oct 30 01:32:51.384245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:51.384032 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs" event={"ID":"f302c26f-43ab-4212-95f0-f00e05a6aca4","Type":"ContainerDied","Data":"9d5924d83de4a1a7aa42c9d5f3274dc03575d23a51418f0c434594ab898b8109"} Oct 30 01:32:51.384245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:32:51.384078 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d5924d83de4a1a7aa42c9d5f3274dc03575d23a51418f0c434594ab898b8109" Oct 30 01:33:09.160490 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.160451 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs"] Oct 30 01:33:09.174282 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.174245 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nhbcs"] Oct 30 01:33:09.191699 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.191663 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f302c26f-43ab-4212-95f0-f00e05a6aca4" path="/var/lib/kubelet/pods/f302c26f-43ab-4212-95f0-f00e05a6aca4/volumes" Oct 30 01:33:09.207253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.207226 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8"] Oct 30 01:33:09.207725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.207708 4905 cpu_manager.go:414] "containerMap: removing container" podUID="f302c26f-43ab-4212-95f0-f00e05a6aca4" containerName="revoke-keys" Oct 30 01:33:09.207788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.207726 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f302c26f-43ab-4212-95f0-f00e05a6aca4" containerName="revoke-keys" Oct 30 01:33:09.207833 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.207803 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="f302c26f-43ab-4212-95f0-f00e05a6aca4" containerName="revoke-keys" Oct 30 01:33:09.211196 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.211180 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.228048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.228016 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:33:09.228048 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.228032 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:33:09.228305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.228150 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:33:09.228305 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.228172 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:33:09.231134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.231103 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8"] Oct 30 01:33:09.350965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.350929 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27hv\" (UniqueName: \"kubernetes.io/projected/d3ae4d67-330d-433a-97d6-fb845bd35174-kube-api-access-v27hv\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.351147 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.351104 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.351194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.351147 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.452376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.452274 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.452376 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.452337 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.452679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.452379 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27hv\" (UniqueName: \"kubernetes.io/projected/d3ae4d67-330d-433a-97d6-fb845bd35174-kube-api-access-v27hv\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.453046 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.453022 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.454919 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.454896 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.473031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.473006 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27hv\" (UniqueName: \"kubernetes.io/projected/d3ae4d67-330d-433a-97d6-fb845bd35174-kube-api-access-v27hv\") pod \"maas-api-revoke-sub-3d4156f1-rpdx8\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.526218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.526190 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:09.823766 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:33:09.823733 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ae4d67_330d_433a_97d6_fb845bd35174.slice/crio-73a63c8da81a06c242823164b78574883eef34bf2b0b2f1e4a69231590352964 WatchSource:0}: Error finding container 73a63c8da81a06c242823164b78574883eef34bf2b0b2f1e4a69231590352964: Status 404 returned error can't find the container with id 73a63c8da81a06c242823164b78574883eef34bf2b0b2f1e4a69231590352964 Oct 30 01:33:09.823889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:09.823809 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8"] Oct 30 01:33:10.518228 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:10.518193 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" event={"ID":"d3ae4d67-330d-433a-97d6-fb845bd35174","Type":"ContainerStarted","Data":"b4d5f4f1b6eed1c8cb6d995760a8151d624777a8a0f5e2071941d719ea7fd6c8"} Oct 30 01:33:10.518687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:10.518246 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" event={"ID":"d3ae4d67-330d-433a-97d6-fb845bd35174","Type":"ContainerStarted","Data":"73a63c8da81a06c242823164b78574883eef34bf2b0b2f1e4a69231590352964"} Oct 30 01:33:10.559765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:10.559714 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" podStartSLOduration=1.559701362 podStartE2EDuration="1.559701362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:10.558933366 +0000 UTC m=+3275.502106572" watchObservedRunningTime="2026-10-30 01:33:10.559701362 +0000 UTC m=+3275.502874591" Oct 30 01:33:11.525264 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:11.525233 4905 generic.go:356] "Generic (PLEG): container finished" podID="d3ae4d67-330d-433a-97d6-fb845bd35174" containerID="b4d5f4f1b6eed1c8cb6d995760a8151d624777a8a0f5e2071941d719ea7fd6c8" exitCode=0 Oct 30 01:33:11.525714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:11.525316 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" event={"ID":"d3ae4d67-330d-433a-97d6-fb845bd35174","Type":"ContainerDied","Data":"b4d5f4f1b6eed1c8cb6d995760a8151d624777a8a0f5e2071941d719ea7fd6c8"} Oct 30 01:33:12.413882 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:33:12.413844 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:33:12.673939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.673884 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:12.785247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.785214 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-xks-ca\") pod \"d3ae4d67-330d-433a-97d6-fb845bd35174\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " Oct 30 01:33:12.785420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.785352 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-openshift-ca\") pod \"d3ae4d67-330d-433a-97d6-fb845bd35174\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " Oct 30 01:33:12.785420 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.785382 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ae4d67-330d-433a-97d6-fb845bd35174-kube-api-access-v27hv\" (UniqueName: \"kubernetes.io/projected/d3ae4d67-330d-433a-97d6-fb845bd35174-kube-api-access-v27hv\") pod \"d3ae4d67-330d-433a-97d6-fb845bd35174\" (UID: \"d3ae4d67-330d-433a-97d6-fb845bd35174\") " Oct 30 01:33:12.785746 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.785715 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-openshift-ca" pod "d3ae4d67-330d-433a-97d6-fb845bd35174" (UID: "d3ae4d67-330d-433a-97d6-fb845bd35174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:12.787610 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.787582 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-xks-ca" pod "d3ae4d67-330d-433a-97d6-fb845bd35174" (UID: "d3ae4d67-330d-433a-97d6-fb845bd35174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:12.787732 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.787715 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ae4d67-330d-433a-97d6-fb845bd35174-kube-api-access-v27hv" pod "d3ae4d67-330d-433a-97d6-fb845bd35174" (UID: "d3ae4d67-330d-433a-97d6-fb845bd35174"). InnerVolumeSpecName "kube-api-access-v27hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:12.886361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.886329 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:33:12.886361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.886354 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v27hv\" (UniqueName: \"kubernetes.io/projected/d3ae4d67-330d-433a-97d6-fb845bd35174-kube-api-access-v27hv\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:33:12.886361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:12.886365 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d67-330d-433a-97d6-fb845bd35174-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:33:13.540697 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:13.540670 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" Oct 30 01:33:13.540697 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:13.540678 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8" event={"ID":"d3ae4d67-330d-433a-97d6-fb845bd35174","Type":"ContainerDied","Data":"73a63c8da81a06c242823164b78574883eef34bf2b0b2f1e4a69231590352964"} Oct 30 01:33:13.540965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:13.540723 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73a63c8da81a06c242823164b78574883eef34bf2b0b2f1e4a69231590352964" Oct 30 01:33:31.035009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:31.034974 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj"] Oct 30 01:33:31.048351 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:31.048305 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127241af-scgpj"] Oct 30 01:33:31.189712 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:31.189676 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af47d61-5943-430a-9a1b-a8f7f12a952a" path="/var/lib/kubelet/pods/5af47d61-5943-430a-9a1b-a8f7f12a952a/volumes" Oct 30 01:33:32.719301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.719271 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8"] Oct 30 01:33:32.732478 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.732451 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpdx8"] Oct 30 01:33:32.759993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.759965 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v"] Oct 30 01:33:32.760403 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.760391 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d3ae4d67-330d-433a-97d6-fb845bd35174" containerName="revoke-keys" Oct 30 01:33:32.760459 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.760404 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ae4d67-330d-433a-97d6-fb845bd35174" containerName="revoke-keys" Oct 30 01:33:32.760502 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.760490 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ae4d67-330d-433a-97d6-fb845bd35174" containerName="revoke-keys" Oct 30 01:33:32.764982 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.764965 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.782665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.782631 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:33:32.782665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.782646 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:33:32.782885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.782672 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:33:32.782885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.782710 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:33:32.784885 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.784864 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v"] Oct 30 01:33:32.894603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.894570 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthj5\" (UniqueName: \"kubernetes.io/projected/7f2eef3c-0167-4722-908d-47f2d286ab86-kube-api-access-fthj5\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.894788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.894616 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.894788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.894735 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.995379 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.995344 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.995614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.995457 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthj5\" (UniqueName: \"kubernetes.io/projected/7f2eef3c-0167-4722-908d-47f2d286ab86-kube-api-access-fthj5\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.995614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.995478 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.996088 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.996067 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:32.997944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:32.997923 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:33.023597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:33.023565 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthj5\" (UniqueName: \"kubernetes.io/projected/7f2eef3c-0167-4722-908d-47f2d286ab86-kube-api-access-fthj5\") pod \"maas-api-revoke-sub-3d4156f1-9464v\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:33.079029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:33.078989 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:33.189177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:33.189142 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ae4d67-330d-433a-97d6-fb845bd35174" path="/var/lib/kubelet/pods/d3ae4d67-330d-433a-97d6-fb845bd35174/volumes" Oct 30 01:33:33.373130 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:33:33.373102 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2eef3c_0167_4722_908d_47f2d286ab86.slice/crio-cc05b24fa02decb447250801eec13715bf1f04168a57da135cdc05ab392e84d1 WatchSource:0}: Error finding container cc05b24fa02decb447250801eec13715bf1f04168a57da135cdc05ab392e84d1: Status 404 returned error can't find the container with id cc05b24fa02decb447250801eec13715bf1f04168a57da135cdc05ab392e84d1 Oct 30 01:33:33.374393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:33.374360 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v"] Oct 30 01:33:33.654946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:33.654910 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" event={"ID":"7f2eef3c-0167-4722-908d-47f2d286ab86","Type":"ContainerStarted","Data":"bece83a414c7aed065fe42ee6bc85efb9e236fbfff2b316bf1ae5d6809abf6a8"} Oct 30 01:33:33.655120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:33.654969 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" event={"ID":"7f2eef3c-0167-4722-908d-47f2d286ab86","Type":"ContainerStarted","Data":"cc05b24fa02decb447250801eec13715bf1f04168a57da135cdc05ab392e84d1"} Oct 30 01:33:33.697336 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:33.697283 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" podStartSLOduration=1.69726635 podStartE2EDuration="1.69726635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:33.694999164 +0000 UTC m=+3298.638172357" watchObservedRunningTime="2026-10-30 01:33:33.69726635 +0000 UTC m=+3298.640439542" Oct 30 01:33:34.662619 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:34.662584 4905 generic.go:356] "Generic (PLEG): container finished" podID="7f2eef3c-0167-4722-908d-47f2d286ab86" containerID="bece83a414c7aed065fe42ee6bc85efb9e236fbfff2b316bf1ae5d6809abf6a8" exitCode=0 Oct 30 01:33:34.663032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:34.662628 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" event={"ID":"7f2eef3c-0167-4722-908d-47f2d286ab86","Type":"ContainerDied","Data":"bece83a414c7aed065fe42ee6bc85efb9e236fbfff2b316bf1ae5d6809abf6a8"} Oct 30 01:33:35.807766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:35.807739 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:35.927586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:35.927494 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-openshift-ca\") pod \"7f2eef3c-0167-4722-908d-47f2d286ab86\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " Oct 30 01:33:35.927586 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:35.927546 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-xks-ca\") pod \"7f2eef3c-0167-4722-908d-47f2d286ab86\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " Oct 30 01:33:35.927850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:35.927658 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2eef3c-0167-4722-908d-47f2d286ab86-kube-api-access-fthj5\" (UniqueName: \"kubernetes.io/projected/7f2eef3c-0167-4722-908d-47f2d286ab86-kube-api-access-fthj5\") pod \"7f2eef3c-0167-4722-908d-47f2d286ab86\" (UID: \"7f2eef3c-0167-4722-908d-47f2d286ab86\") " Oct 30 01:33:35.927916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:35.927855 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-openshift-ca" pod "7f2eef3c-0167-4722-908d-47f2d286ab86" (UID: "7f2eef3c-0167-4722-908d-47f2d286ab86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:35.929905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:35.929878 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-xks-ca" pod "7f2eef3c-0167-4722-908d-47f2d286ab86" (UID: "7f2eef3c-0167-4722-908d-47f2d286ab86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:35.930005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:35.929924 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2eef3c-0167-4722-908d-47f2d286ab86-kube-api-access-fthj5" pod "7f2eef3c-0167-4722-908d-47f2d286ab86" (UID: "7f2eef3c-0167-4722-908d-47f2d286ab86"). InnerVolumeSpecName "kube-api-access-fthj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:36.029327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.029292 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:33:36.029327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.029324 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2eef3c-0167-4722-908d-47f2d286ab86-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:33:36.029327 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.029333 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthj5\" (UniqueName: \"kubernetes.io/projected/7f2eef3c-0167-4722-908d-47f2d286ab86-kube-api-access-fthj5\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:33:36.545431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.545403 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:33:36.547200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.547172 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:33:36.675842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.675811 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" Oct 30 01:33:36.676030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.675844 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v" event={"ID":"7f2eef3c-0167-4722-908d-47f2d286ab86","Type":"ContainerDied","Data":"cc05b24fa02decb447250801eec13715bf1f04168a57da135cdc05ab392e84d1"} Oct 30 01:33:36.676030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:36.675893 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc05b24fa02decb447250801eec13715bf1f04168a57da135cdc05ab392e84d1" Oct 30 01:33:42.413424 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:33:42.413387 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:33:47.145552 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:47.145508 4905 scope.go:122] "RemoveContainer" containerID="bf415f8d07d2c62636b59cc58818708768edcd876079f448428522e0fc2a2691" Oct 30 01:33:54.514799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.514754 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s"] Oct 30 01:33:54.515489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.515459 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7f2eef3c-0167-4722-908d-47f2d286ab86" containerName="revoke-keys" Oct 30 01:33:54.515489 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.515488 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2eef3c-0167-4722-908d-47f2d286ab86" containerName="revoke-keys" Oct 30 01:33:54.515748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.515630 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2eef3c-0167-4722-908d-47f2d286ab86" containerName="revoke-keys" Oct 30 01:33:54.521473 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.521448 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.537229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.537206 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-364d04-openshift-default-dockercfg-zcdbv\"" Oct 30 01:33:54.554993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.554963 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s"] Oct 30 01:33:54.611544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.611482 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae28e6e-e746-4576-92db-9210c7f39174-istio-podinfo\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.611702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.611569 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-socket\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.611702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.611672 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-credential-socket\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.611784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.611720 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-istio-token\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.611784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.611740 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-envoy\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.611899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.611870 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae28e6e-e746-4576-92db-9210c7f39174-istiod-ca-cert\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.611993 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.611967 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-data\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.612131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.612030 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-certs\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.612131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.612109 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshdw\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-kube-api-access-jshdw\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.713627 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.713593 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-credential-socket\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.713815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.713645 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-istio-token\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.713815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.713769 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-envoy\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.713900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.713819 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae28e6e-e746-4576-92db-9210c7f39174-istiod-ca-cert\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.713900 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.713866 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-data\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.713921 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-certs\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714003 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.713963 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshdw\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-kube-api-access-jshdw\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.714013 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae28e6e-e746-4576-92db-9210c7f39174-istio-podinfo\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.714027 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-credential-socket\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.714048 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-socket\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.714266 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-data\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.714289 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-certs\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.714502 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-socket\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.714787 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.714661 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae28e6e-e746-4576-92db-9210c7f39174-istiod-ca-cert\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.716318 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.716301 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-envoy\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.716710 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.716691 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae28e6e-e746-4576-92db-9210c7f39174-istio-podinfo\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.740366 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.740336 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-istio-token\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.740499 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.740400 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshdw\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-kube-api-access-jshdw\") pod \"e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:54.836613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:54.836504 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:55.146418 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:33:55.146375 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae28e6e_e746_4576_92db_9210c7f39174.slice/crio-cd2a042eba8d673f15614dc94d1fef08758648948dea95b13c5539e96b2be77f WatchSource:0}: Error finding container cd2a042eba8d673f15614dc94d1fef08758648948dea95b13c5539e96b2be77f: Status 404 returned error can't find the container with id cd2a042eba8d673f15614dc94d1fef08758648948dea95b13c5539e96b2be77f Oct 30 01:33:55.147084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.147012 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s"] Oct 30 01:33:55.148776 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.148744 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:33:55.148868 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.148820 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:33:55.148940 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.148866 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:33:55.796594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.796559 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" event={"ID":"aae28e6e-e746-4576-92db-9210c7f39174","Type":"ContainerStarted","Data":"21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105"} Oct 30 01:33:55.797122 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.796609 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" event={"ID":"aae28e6e-e746-4576-92db-9210c7f39174","Type":"ContainerStarted","Data":"cd2a042eba8d673f15614dc94d1fef08758648948dea95b13c5539e96b2be77f"} Oct 30 01:33:55.837127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.837101 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:55.855764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:55.855721 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podStartSLOduration=1.855708467 podStartE2EDuration="1.855708467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:55.853604024 +0000 UTC m=+3320.796777230" watchObservedRunningTime="2026-10-30 01:33:55.855708467 +0000 UTC m=+3320.798881657" Oct 30 01:33:56.842858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:56.842825 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:57.808455 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:57.808423 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:33:57.809574 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:33:57.809512 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:34:00.019320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.019282 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f"] Oct 30 01:34:00.023428 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.023405 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.046025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.046000 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f"] Oct 30 01:34:00.070012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.056030 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-364d04\"" Oct 30 01:34:00.145944 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.145907 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6bc79248-dockercfg-7nr7n\"" Oct 30 01:34:00.154086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.152579 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596"] Oct 30 01:34:00.157508 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.157482 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596"] Oct 30 01:34:00.157679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.157615 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.171772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.171739 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/9af75b68-2c68-4634-984a-51e174aa604a-kube-api-access-p4xqk\") pod \"payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.171941 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.171874 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af75b68-2c68-4634-984a-51e174aa604a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.272685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.272599 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thmw\" (UniqueName: \"kubernetes.io/projected/6e2a5457-a145-4275-a212-af8b51930056-kube-api-access-4thmw\") pod \"payload-processing-e2e-worker-single-main-364d04-d4c64b96652596\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.272685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.272669 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a5457-a145-4275-a212-af8b51930056-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-364d04-d4c64b96652596\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.272877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.272771 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/9af75b68-2c68-4634-984a-51e174aa604a-kube-api-access-p4xqk\") pod \"payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.272877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.272843 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af75b68-2c68-4634-984a-51e174aa604a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.273562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.273544 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af75b68-2c68-4634-984a-51e174aa604a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.294191 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.294165 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/9af75b68-2c68-4634-984a-51e174aa604a-kube-api-access-p4xqk\") pod \"payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.337165 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.337134 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.373764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.373733 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thmw\" (UniqueName: \"kubernetes.io/projected/6e2a5457-a145-4275-a212-af8b51930056-kube-api-access-4thmw\") pod \"payload-processing-e2e-worker-single-main-364d04-d4c64b96652596\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.373935 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.373797 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a5457-a145-4275-a212-af8b51930056-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-364d04-d4c64b96652596\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.374577 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.374541 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a5457-a145-4275-a212-af8b51930056-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-364d04-d4c64b96652596\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.399749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.399691 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thmw\" (UniqueName: \"kubernetes.io/projected/6e2a5457-a145-4275-a212-af8b51930056-kube-api-access-4thmw\") pod \"payload-processing-e2e-worker-single-main-364d04-d4c64b96652596\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.471535 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.471497 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:00.635368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.635336 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f"] Oct 30 01:34:00.636009 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:34:00.635984 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af75b68_2c68_4634_984a_51e174aa604a.slice/crio-4db5b02e647946cfbe512b429a00b61baaaa7a43e618b1ea4fcc1cc4d3273113 WatchSource:0}: Error finding container 4db5b02e647946cfbe512b429a00b61baaaa7a43e618b1ea4fcc1cc4d3273113: Status 404 returned error can't find the container with id 4db5b02e647946cfbe512b429a00b61baaaa7a43e618b1ea4fcc1cc4d3273113 Oct 30 01:34:00.765486 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:34:00.765455 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2a5457_a145_4275_a212_af8b51930056.slice/crio-cbaa2c0a8ad694b5a675bc0427feed6b4a912235b5e8d75c5d5ac440391a43b8 WatchSource:0}: Error finding container cbaa2c0a8ad694b5a675bc0427feed6b4a912235b5e8d75c5d5ac440391a43b8: Status 404 returned error can't find the container with id cbaa2c0a8ad694b5a675bc0427feed6b4a912235b5e8d75c5d5ac440391a43b8 Oct 30 01:34:00.765970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.765942 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596"] Oct 30 01:34:00.826675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.826257 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" event={"ID":"9af75b68-2c68-4634-984a-51e174aa604a","Type":"ContainerStarted","Data":"3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03"} Oct 30 01:34:00.826675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.826331 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" event={"ID":"9af75b68-2c68-4634-984a-51e174aa604a","Type":"ContainerStarted","Data":"4db5b02e647946cfbe512b429a00b61baaaa7a43e618b1ea4fcc1cc4d3273113"} Oct 30 01:34:00.826675 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.826460 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:00.828371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.828351 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" event={"ID":"6e2a5457-a145-4275-a212-af8b51930056","Type":"ContainerStarted","Data":"cbaa2c0a8ad694b5a675bc0427feed6b4a912235b5e8d75c5d5ac440391a43b8"} Oct 30 01:34:00.873766 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:00.873716 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" podStartSLOduration=1.87370101 podStartE2EDuration="1.87370101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:00.870951695 +0000 UTC m=+3325.814124889" watchObservedRunningTime="2026-10-30 01:34:00.87370101 +0000 UTC m=+3325.816874200" Oct 30 01:34:01.835641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:01.835599 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" event={"ID":"6e2a5457-a145-4275-a212-af8b51930056","Type":"ContainerStarted","Data":"5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be"} Oct 30 01:34:01.836178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:01.835709 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:01.876884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:01.876829 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" podStartSLOduration=1.8768148249999999 podStartE2EDuration="1.876814825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:01.876299824 +0000 UTC m=+3326.819473017" watchObservedRunningTime="2026-10-30 01:34:01.876814825 +0000 UTC m=+3326.819988017" Oct 30 01:34:11.839308 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:11.839272 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:34:12.414548 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:34:12.414495 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:34:12.810065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.810031 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Oct 30 01:34:12.810277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.810087 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Oct 30 01:34:12.844810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.844782 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:34:12.926334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.926258 4905 generic.go:356] "Generic (PLEG): container finished" podID="aae28e6e-e746-4576-92db-9210c7f39174" containerID="21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105" exitCode=0 Oct 30 01:34:12.926334 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.926304 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" event={"ID":"aae28e6e-e746-4576-92db-9210c7f39174","Type":"ContainerDied","Data":"21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105"} Oct 30 01:34:12.926733 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.926718 4905 scope.go:122] "RemoveContainer" containerID="21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105" Oct 30 01:34:12.927960 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.927924 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:34:12.928059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.928001 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:34:12.928059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:12.928044 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:34:13.956412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:13.956377 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" event={"ID":"aae28e6e-e746-4576-92db-9210c7f39174","Type":"ContainerStarted","Data":"f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9"} Oct 30 01:34:14.845892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:14.845859 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:34:14.847030 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:14.846195 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Oct 30 01:34:14.847219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:14.847194 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Oct 30 01:34:15.841546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:15.839906 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Oct 30 01:34:15.841546 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:15.839981 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Oct 30 01:34:16.844545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:16.841917 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Oct 30 01:34:16.844545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:16.841979 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Oct 30 01:34:17.840954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:17.840912 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Oct 30 01:34:17.841304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:17.841275 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Oct 30 01:34:18.846543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:18.845869 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Oct 30 01:34:18.846543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:18.845929 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Oct 30 01:34:19.025532 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:19.025470 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt"] Oct 30 01:34:19.039131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:19.039096 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ml9pt"] Oct 30 01:34:19.198156 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:19.198075 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f95758d-68d4-44cc-84d9-a8a13e1a363d" path="/var/lib/kubelet/pods/0f95758d-68d4-44cc-84d9-a8a13e1a363d/volumes" Oct 30 01:34:19.841229 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:19.841200 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:34:19.841737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:19.841719 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:34:19.842319 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:19.842302 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:34:38.537192 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.537154 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc"] Oct 30 01:34:38.541075 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.541058 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.571839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.571804 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:38.571997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.571868 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:38.571997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.571876 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:34:38.571997 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.571887 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:38.574449 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.574419 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc"] Oct 30 01:34:38.631606 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.631575 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.631745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.631623 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4kn\" (UniqueName: \"kubernetes.io/projected/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-kube-api-access-gq4kn\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.631745 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.631707 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.732836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.732808 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.732992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.732867 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4kn\" (UniqueName: \"kubernetes.io/projected/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-kube-api-access-gq4kn\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.732992 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.732895 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.733667 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.733506 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.735378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.735352 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.753838 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.753815 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4kn\" (UniqueName: \"kubernetes.io/projected/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-kube-api-access-gq4kn\") pod \"maas-api-revoke-sub-58e54186-prxhc\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:38.853718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:38.853658 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:39.148372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:39.148343 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc"] Oct 30 01:34:39.148826 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:34:39.148796 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb5a0d6_b402_4d35_92e8_2f63681b34d8.slice/crio-cbbd0c1004f6ac48bca5430ba43a55c6533a2552d305c5ba06af1f53c8e17773 WatchSource:0}: Error finding container cbbd0c1004f6ac48bca5430ba43a55c6533a2552d305c5ba06af1f53c8e17773: Status 404 returned error can't find the container with id cbbd0c1004f6ac48bca5430ba43a55c6533a2552d305c5ba06af1f53c8e17773 Oct 30 01:34:40.121277 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:40.121240 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" event={"ID":"0eb5a0d6-b402-4d35-92e8-2f63681b34d8","Type":"ContainerStarted","Data":"7f76225f62323bd9e80b023f1844f84311f656385b321a2c47dab32efd634625"} Oct 30 01:34:40.121806 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:40.121287 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" event={"ID":"0eb5a0d6-b402-4d35-92e8-2f63681b34d8","Type":"ContainerStarted","Data":"cbbd0c1004f6ac48bca5430ba43a55c6533a2552d305c5ba06af1f53c8e17773"} Oct 30 01:34:40.167170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:40.167121 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" podStartSLOduration=2.167107283 podStartE2EDuration="2.167107283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:40.165098277 +0000 UTC m=+3365.108271470" watchObservedRunningTime="2026-10-30 01:34:40.167107283 +0000 UTC m=+3365.110280476" Oct 30 01:34:41.129123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:41.129091 4905 generic.go:356] "Generic (PLEG): container finished" podID="0eb5a0d6-b402-4d35-92e8-2f63681b34d8" containerID="7f76225f62323bd9e80b023f1844f84311f656385b321a2c47dab32efd634625" exitCode=0 Oct 30 01:34:41.129550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:41.129154 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" event={"ID":"0eb5a0d6-b402-4d35-92e8-2f63681b34d8","Type":"ContainerDied","Data":"7f76225f62323bd9e80b023f1844f84311f656385b321a2c47dab32efd634625"} Oct 30 01:34:42.276296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.276267 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:42.378120 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.378087 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-xks-ca\") pod \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " Oct 30 01:34:42.378294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.378164 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-openshift-ca\") pod \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " Oct 30 01:34:42.378294 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.378273 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-kube-api-access-gq4kn\" (UniqueName: \"kubernetes.io/projected/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-kube-api-access-gq4kn\") pod \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\" (UID: \"0eb5a0d6-b402-4d35-92e8-2f63681b34d8\") " Oct 30 01:34:42.378583 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.378556 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-openshift-ca" pod "0eb5a0d6-b402-4d35-92e8-2f63681b34d8" (UID: "0eb5a0d6-b402-4d35-92e8-2f63681b34d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:42.380462 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.380442 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-xks-ca" pod "0eb5a0d6-b402-4d35-92e8-2f63681b34d8" (UID: "0eb5a0d6-b402-4d35-92e8-2f63681b34d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:42.380595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.380572 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-kube-api-access-gq4kn" pod "0eb5a0d6-b402-4d35-92e8-2f63681b34d8" (UID: "0eb5a0d6-b402-4d35-92e8-2f63681b34d8"). InnerVolumeSpecName "kube-api-access-gq4kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:42.413508 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:34:42.413470 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:34:42.479464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.479434 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:34:42.479464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.479458 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:34:42.479464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:42.479468 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4kn\" (UniqueName: \"kubernetes.io/projected/0eb5a0d6-b402-4d35-92e8-2f63681b34d8-kube-api-access-gq4kn\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:34:43.140647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:43.140611 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" event={"ID":"0eb5a0d6-b402-4d35-92e8-2f63681b34d8","Type":"ContainerDied","Data":"cbbd0c1004f6ac48bca5430ba43a55c6533a2552d305c5ba06af1f53c8e17773"} Oct 30 01:34:43.140647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:43.140644 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc" Oct 30 01:34:43.140869 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:43.140672 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbbd0c1004f6ac48bca5430ba43a55c6533a2552d305c5ba06af1f53c8e17773" Oct 30 01:34:47.170213 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:47.170139 4905 scope.go:122] "RemoveContainer" containerID="69741ac64a4f5321d9da12ed5635c20132364cc40b8a1dc94a3ebf72ee1f2fcd" Oct 30 01:34:47.179400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:47.179377 4905 scope.go:122] "RemoveContainer" containerID="d32551c9b734dc710f92e1887971316a330b124ffeda442132653924adc5e090" Oct 30 01:34:47.187815 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:47.187792 4905 scope.go:122] "RemoveContainer" containerID="383406d1ae8c80d61f5d52d5dc5a1a1ae425568e35d12a56216d166a2fd47077" Oct 30 01:34:49.666174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.666132 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw"] Oct 30 01:34:49.666744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.666634 4905 cpu_manager.go:414] "containerMap: removing container" podUID="0eb5a0d6-b402-4d35-92e8-2f63681b34d8" containerName="revoke-keys" Oct 30 01:34:49.666744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.666647 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb5a0d6-b402-4d35-92e8-2f63681b34d8" containerName="revoke-keys" Oct 30 01:34:49.666744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.666744 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb5a0d6-b402-4d35-92e8-2f63681b34d8" containerName="revoke-keys" Oct 30 01:34:49.671824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.671797 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.689908 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.689878 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw"] Oct 30 01:34:49.700310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.700276 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:49.700310 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.700299 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:49.700491 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.700299 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:49.701557 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.701540 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:34:49.751939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.751897 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.752125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.752067 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxqz\" (UniqueName: \"kubernetes.io/projected/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-kube-api-access-jpxqz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.752201 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.752183 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.856542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.853183 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.856542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.853311 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxqz\" (UniqueName: \"kubernetes.io/projected/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-kube-api-access-jpxqz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.856542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.853382 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.856542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.854301 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.864130 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.864094 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.876775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.876746 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxqz\" (UniqueName: \"kubernetes.io/projected/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-kube-api-access-jpxqz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:49.986773 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:49.986743 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:50.289360 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:34:50.289326 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46983f9b_dad2_4b4e_8b2e_f66a3c43ec3f.slice/crio-25c354cbcaf34e6b105901a4e707b363b1f0dfb9abd8f60746d3dcb8c3b005c4 WatchSource:0}: Error finding container 25c354cbcaf34e6b105901a4e707b363b1f0dfb9abd8f60746d3dcb8c3b005c4: Status 404 returned error can't find the container with id 25c354cbcaf34e6b105901a4e707b363b1f0dfb9abd8f60746d3dcb8c3b005c4 Oct 30 01:34:50.289572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:50.289551 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw"] Oct 30 01:34:51.188139 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:51.188110 4905 generic.go:356] "Generic (PLEG): container finished" podID="46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" containerID="07e8a62c48d1064eb59f7f09b465f0a9319f950f3d82093ebcb329812a03a62e" exitCode=0 Oct 30 01:34:51.189207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:51.189184 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" event={"ID":"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f","Type":"ContainerDied","Data":"07e8a62c48d1064eb59f7f09b465f0a9319f950f3d82093ebcb329812a03a62e"} Oct 30 01:34:51.189275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:51.189225 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" event={"ID":"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f","Type":"ContainerStarted","Data":"25c354cbcaf34e6b105901a4e707b363b1f0dfb9abd8f60746d3dcb8c3b005c4"} Oct 30 01:34:52.343644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.343616 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:52.375723 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.375695 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-kube-api-access-jpxqz\" (UniqueName: \"kubernetes.io/projected/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-kube-api-access-jpxqz\") pod \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " Oct 30 01:34:52.375889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.375735 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-openshift-ca\") pod \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " Oct 30 01:34:52.376124 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.376104 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-openshift-ca" pod "46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" (UID: "46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:52.377957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.377932 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-kube-api-access-jpxqz" pod "46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" (UID: "46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f"). InnerVolumeSpecName "kube-api-access-jpxqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:52.477281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.477196 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-xks-ca\") pod \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\" (UID: \"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f\") " Oct 30 01:34:52.477485 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.477472 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpxqz\" (UniqueName: \"kubernetes.io/projected/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-kube-api-access-jpxqz\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:34:52.477571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.477490 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:34:52.479436 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.479402 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-xks-ca" pod "46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" (UID: "46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:52.578150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:52.578119 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:34:53.203004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:53.202971 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" Oct 30 01:34:53.203187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:53.202998 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw" event={"ID":"46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f","Type":"ContainerDied","Data":"25c354cbcaf34e6b105901a4e707b363b1f0dfb9abd8f60746d3dcb8c3b005c4"} Oct 30 01:34:53.203187 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:34:53.203098 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c354cbcaf34e6b105901a4e707b363b1f0dfb9abd8f60746d3dcb8c3b005c4" Oct 30 01:35:00.099858 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:00.099816 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596"] Oct 30 01:35:00.134757 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:00.134729 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f"] Oct 30 01:35:02.600636 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:02.600600 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw"] Oct 30 01:35:02.622175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:02.621729 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-364d04-9hgtw"] Oct 30 01:35:03.192396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:03.192353 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" path="/var/lib/kubelet/pods/46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f/volumes" Oct 30 01:35:05.101025 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:05.100984 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" podUID="6e2a5457-a145-4275-a212-af8b51930056" containerName="payload-processing" containerID="cri-o://5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be" gracePeriod=25 Oct 30 01:35:05.135373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:05.135307 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" podUID="9af75b68-2c68-4634-984a-51e174aa604a" containerName="payload-pre-processing" containerID="cri-o://3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03" gracePeriod=25 Oct 30 01:35:09.068267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:09.068223 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s"] Oct 30 01:35:09.068824 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:09.068596 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" containerID="cri-o://f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9" gracePeriod=30 Oct 30 01:35:10.240116 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.240077 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v"] Oct 30 01:35:10.253450 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.253422 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9464v"] Oct 30 01:35:10.286042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.286011 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h"] Oct 30 01:35:10.286495 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.286481 4905 cpu_manager.go:414] "containerMap: removing container" podUID="46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" containerName="revoke-keys" Oct 30 01:35:10.286598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.286497 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" containerName="revoke-keys" Oct 30 01:35:10.286645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.286627 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="46983f9b-dad2-4b4e-8b2e-f66a3c43ec3f" containerName="revoke-keys" Oct 30 01:35:10.290081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.290061 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.306587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.306556 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:35:10.306777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.306639 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:35:10.306777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.306661 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:35:10.306777 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.306677 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:35:10.310350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.310324 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h"] Oct 30 01:35:10.366774 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.366743 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mct74\" (UniqueName: \"kubernetes.io/projected/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-kube-api-access-mct74\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.366927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.366798 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.366927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.366897 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.468016 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.467976 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mct74\" (UniqueName: \"kubernetes.io/projected/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-kube-api-access-mct74\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.468259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.468058 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.468259 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.468110 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.468905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.468876 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.470614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.470593 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.488797 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.488772 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mct74\" (UniqueName: \"kubernetes.io/projected/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-kube-api-access-mct74\") pod \"maas-api-revoke-sub-3d4156f1-4s27h\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.603814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.603722 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:10.909694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:10.909661 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h"] Oct 30 01:35:10.910254 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:35:10.910213 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e00a2a3_e8c3_4ea8_9a7e_5828ca03f592.slice/crio-a0e684e22154813edd4c4c59e0c260ed88a0afca2d9970bf5b6f1a72f1657ad1 WatchSource:0}: Error finding container a0e684e22154813edd4c4c59e0c260ed88a0afca2d9970bf5b6f1a72f1657ad1: Status 404 returned error can't find the container with id a0e684e22154813edd4c4c59e0c260ed88a0afca2d9970bf5b6f1a72f1657ad1 Oct 30 01:35:11.189755 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:11.189673 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2eef3c-0167-4722-908d-47f2d286ab86" path="/var/lib/kubelet/pods/7f2eef3c-0167-4722-908d-47f2d286ab86/volumes" Oct 30 01:35:11.313439 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:11.313402 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" event={"ID":"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592","Type":"ContainerStarted","Data":"127b7b3a8a3185266821945303ae4cb97715dcb004692f3643c6544d43214162"} Oct 30 01:35:11.313831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:11.313458 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" event={"ID":"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592","Type":"ContainerStarted","Data":"a0e684e22154813edd4c4c59e0c260ed88a0afca2d9970bf5b6f1a72f1657ad1"} Oct 30 01:35:11.356404 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:11.356359 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" podStartSLOduration=1.356345949 podStartE2EDuration="1.356345949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:11.354302804 +0000 UTC m=+3396.297476007" watchObservedRunningTime="2026-10-30 01:35:11.356345949 +0000 UTC m=+3396.299519141" Oct 30 01:35:11.837481 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:11.837444 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" podUID="9af75b68-2c68-4634-984a-51e174aa604a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:35:12.320845 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:12.320815 4905 generic.go:356] "Generic (PLEG): container finished" podID="4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" containerID="127b7b3a8a3185266821945303ae4cb97715dcb004692f3643c6544d43214162" exitCode=0 Oct 30 01:35:12.321219 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:12.320907 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" event={"ID":"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592","Type":"ContainerDied","Data":"127b7b3a8a3185266821945303ae4cb97715dcb004692f3643c6544d43214162"} Oct 30 01:35:12.415255 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:35:12.415218 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:35:12.843059 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:12.843026 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" podUID="6e2a5457-a145-4275-a212-af8b51930056" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:35:13.462186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.462160 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:13.600127 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.600037 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-openshift-ca\") pod \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " Oct 30 01:35:13.600298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.600187 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-xks-ca\") pod \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " Oct 30 01:35:13.600298 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.600209 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-kube-api-access-mct74\" (UniqueName: \"kubernetes.io/projected/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-kube-api-access-mct74\") pod \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\" (UID: \"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592\") " Oct 30 01:35:13.600426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.600389 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-openshift-ca" pod "4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" (UID: "4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:13.602468 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.602442 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-xks-ca" pod "4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" (UID: "4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:13.602589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.602471 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-kube-api-access-mct74" pod "4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" (UID: "4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592"). InnerVolumeSpecName "kube-api-access-mct74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:13.701753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.701719 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:13.701753 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.701751 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:13.701933 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:13.701766 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mct74\" (UniqueName: \"kubernetes.io/projected/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592-kube-api-access-mct74\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.316233 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.316207 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:35:14.333701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.333640 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" Oct 30 01:35:14.333701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.333650 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h" event={"ID":"4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592","Type":"ContainerDied","Data":"a0e684e22154813edd4c4c59e0c260ed88a0afca2d9970bf5b6f1a72f1657ad1"} Oct 30 01:35:14.333701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.333697 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e684e22154813edd4c4c59e0c260ed88a0afca2d9970bf5b6f1a72f1657ad1" Oct 30 01:35:14.335426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.335403 4905 generic.go:356] "Generic (PLEG): container finished" podID="aae28e6e-e746-4576-92db-9210c7f39174" containerID="f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9" exitCode=0 Oct 30 01:35:14.335559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.335451 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" Oct 30 01:35:14.335559 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.335489 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" event={"ID":"aae28e6e-e746-4576-92db-9210c7f39174","Type":"ContainerDied","Data":"f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9"} Oct 30 01:35:14.335661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.335571 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s" event={"ID":"aae28e6e-e746-4576-92db-9210c7f39174","Type":"ContainerDied","Data":"cd2a042eba8d673f15614dc94d1fef08758648948dea95b13c5539e96b2be77f"} Oct 30 01:35:14.335661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.335602 4905 scope.go:122] "RemoveContainer" containerID="f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9" Oct 30 01:35:14.345248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.345231 4905 scope.go:122] "RemoveContainer" containerID="21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105" Oct 30 01:35:14.354296 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.354275 4905 scope.go:122] "RemoveContainer" containerID="f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9" Oct 30 01:35:14.354555 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:35:14.354537 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9\": container with ID starting with f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9 not found: ID does not exist" containerID="f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9" Oct 30 01:35:14.354625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.354564 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9"} err="failed to get container status \"f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9\": rpc error: code = NotFound desc = could not find container \"f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9\": container with ID starting with f456994847d8fbede267df1d1c6deff8b2e6155c995c17cb64221cefa7f82de9 not found: ID does not exist" Oct 30 01:35:14.354625 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.354579 4905 scope.go:122] "RemoveContainer" containerID="21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105" Oct 30 01:35:14.354813 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:35:14.354792 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105\": container with ID starting with 21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105 not found: ID does not exist" containerID="21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105" Oct 30 01:35:14.354860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.354821 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105"} err="failed to get container status \"21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105\": rpc error: code = NotFound desc = could not find container \"21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105\": container with ID starting with 21595e45b3082be0df7be05603391c382bfefd4a43761df56f8166e860386105 not found: ID does not exist" Oct 30 01:35:14.408444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408414 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-socket\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408452 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae28e6e-e746-4576-92db-9210c7f39174-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae28e6e-e746-4576-92db-9210c7f39174-istiod-ca-cert\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408475 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aae28e6e-e746-4576-92db-9210c7f39174-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae28e6e-e746-4576-92db-9210c7f39174-istio-podinfo\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408607 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-credential-socket\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408648 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-certs\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408695 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-data\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408741 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-envoy\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408737 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-socket" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:14.408821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408780 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-istio-token\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-istio-token\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.408821 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.408817 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-kube-api-access-jshdw\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-kube-api-access-jshdw\") pod \"aae28e6e-e746-4576-92db-9210c7f39174\" (UID: \"aae28e6e-e746-4576-92db-9210c7f39174\") " Oct 30 01:35:14.409418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.409158 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.409418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.409243 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-credential-socket" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:14.409418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.409267 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-certs" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:14.409418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.409302 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae28e6e-e746-4576-92db-9210c7f39174-istiod-ca-cert" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:14.409418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.409320 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-data" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:14.411247 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.411216 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-envoy" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:14.411386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.411372 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-kube-api-access-jshdw" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "kube-api-access-jshdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:14.411575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.411553 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aae28e6e-e746-4576-92db-9210c7f39174-istio-podinfo" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:35:14.411650 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.411556 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-istio-token" pod "aae28e6e-e746-4576-92db-9210c7f39174" (UID: "aae28e6e-e746-4576-92db-9210c7f39174"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:14.510931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510905 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.510931 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510932 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.511279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510941 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.511279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510949 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae28e6e-e746-4576-92db-9210c7f39174-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.511279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510957 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.511279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510965 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshdw\" (UniqueName: \"kubernetes.io/projected/aae28e6e-e746-4576-92db-9210c7f39174-kube-api-access-jshdw\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.511279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510973 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae28e6e-e746-4576-92db-9210c7f39174-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.511279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.510981 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae28e6e-e746-4576-92db-9210c7f39174-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:14.684103 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.684070 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s"] Oct 30 01:35:14.686928 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:14.686908 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-364d04-openshift-default-7fbf8f6d45tzt8s"] Oct 30 01:35:15.189602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:15.189570 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae28e6e-e746-4576-92db-9210c7f39174" path="/var/lib/kubelet/pods/aae28e6e-e746-4576-92db-9210c7f39174/volumes" Oct 30 01:35:21.837384 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:21.837343 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" podUID="9af75b68-2c68-4634-984a-51e174aa604a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:35:22.850647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:22.850606 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" podUID="6e2a5457-a145-4275-a212-af8b51930056" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:35:25.396178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.396149 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:35:25.433835 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.433813 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:35:25.441026 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.440997 4905 generic.go:356] "Generic (PLEG): container finished" podID="9af75b68-2c68-4634-984a-51e174aa604a" containerID="3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03" exitCode=0 Oct 30 01:35:25.441131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.441047 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" Oct 30 01:35:25.441131 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.441081 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" event={"ID":"9af75b68-2c68-4634-984a-51e174aa604a","Type":"ContainerDied","Data":"3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03"} Oct 30 01:35:25.441276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.441140 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f" event={"ID":"9af75b68-2c68-4634-984a-51e174aa604a","Type":"ContainerDied","Data":"4db5b02e647946cfbe512b429a00b61baaaa7a43e618b1ea4fcc1cc4d3273113"} Oct 30 01:35:25.441276 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.441168 4905 scope.go:122] "RemoveContainer" containerID="3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03" Oct 30 01:35:25.443419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.443394 4905 generic.go:356] "Generic (PLEG): container finished" podID="6e2a5457-a145-4275-a212-af8b51930056" containerID="5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be" exitCode=0 Oct 30 01:35:25.443587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.443545 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" event={"ID":"6e2a5457-a145-4275-a212-af8b51930056","Type":"ContainerDied","Data":"5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be"} Oct 30 01:35:25.443680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.443585 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" event={"ID":"6e2a5457-a145-4275-a212-af8b51930056","Type":"ContainerDied","Data":"cbaa2c0a8ad694b5a675bc0427feed6b4a912235b5e8d75c5d5ac440391a43b8"} Oct 30 01:35:25.443680 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.443652 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596" Oct 30 01:35:25.453598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.453579 4905 scope.go:122] "RemoveContainer" containerID="3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03" Oct 30 01:35:25.453900 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:35:25.453877 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03\": container with ID starting with 3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03 not found: ID does not exist" containerID="3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03" Oct 30 01:35:25.453965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.453910 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03"} err="failed to get container status \"3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03\": rpc error: code = NotFound desc = could not find container \"3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03\": container with ID starting with 3001330fd0a4970fad8359c5ea3c0d250cf46dfcca1eeaf9244f0ed4d030fd03 not found: ID does not exist" Oct 30 01:35:25.453965 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.453927 4905 scope.go:122] "RemoveContainer" containerID="5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be" Oct 30 01:35:25.463388 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.463368 4905 scope.go:122] "RemoveContainer" containerID="5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be" Oct 30 01:35:25.463695 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:35:25.463676 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be\": container with ID starting with 5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be not found: ID does not exist" containerID="5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be" Oct 30 01:35:25.463781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.463706 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be"} err="failed to get container status \"5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be\": rpc error: code = NotFound desc = could not find container \"5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be\": container with ID starting with 5eb7e39ec139e57a80b43a3c6d9d7b275d943d2207282f6201e936fa66d5c9be not found: ID does not exist" Oct 30 01:35:25.553180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.553148 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af75b68-2c68-4634-984a-51e174aa604a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af75b68-2c68-4634-984a-51e174aa604a-plugins-config-volume\") pod \"9af75b68-2c68-4634-984a-51e174aa604a\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " Oct 30 01:35:25.553180 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.553184 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2a5457-a145-4275-a212-af8b51930056-kube-api-access-4thmw\" (UniqueName: \"kubernetes.io/projected/6e2a5457-a145-4275-a212-af8b51930056-kube-api-access-4thmw\") pod \"6e2a5457-a145-4275-a212-af8b51930056\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " Oct 30 01:35:25.553385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.553267 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af75b68-2c68-4634-984a-51e174aa604a-kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/9af75b68-2c68-4634-984a-51e174aa604a-kube-api-access-p4xqk\") pod \"9af75b68-2c68-4634-984a-51e174aa604a\" (UID: \"9af75b68-2c68-4634-984a-51e174aa604a\") " Oct 30 01:35:25.553385 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.553330 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2a5457-a145-4275-a212-af8b51930056-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a5457-a145-4275-a212-af8b51930056-plugins-config-volume\") pod \"6e2a5457-a145-4275-a212-af8b51930056\" (UID: \"6e2a5457-a145-4275-a212-af8b51930056\") " Oct 30 01:35:25.553629 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.553604 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af75b68-2c68-4634-984a-51e174aa604a-plugins-config-volume" pod "9af75b68-2c68-4634-984a-51e174aa604a" (UID: "9af75b68-2c68-4634-984a-51e174aa604a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:25.553736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.553708 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af75b68-2c68-4634-984a-51e174aa604a-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:25.553927 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.553900 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2a5457-a145-4275-a212-af8b51930056-plugins-config-volume" pod "6e2a5457-a145-4275-a212-af8b51930056" (UID: "6e2a5457-a145-4275-a212-af8b51930056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:25.555626 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.555602 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af75b68-2c68-4634-984a-51e174aa604a-kube-api-access-p4xqk" pod "9af75b68-2c68-4634-984a-51e174aa604a" (UID: "9af75b68-2c68-4634-984a-51e174aa604a"). InnerVolumeSpecName "kube-api-access-p4xqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:25.555713 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.555661 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2a5457-a145-4275-a212-af8b51930056-kube-api-access-4thmw" pod "6e2a5457-a145-4275-a212-af8b51930056" (UID: "6e2a5457-a145-4275-a212-af8b51930056"). InnerVolumeSpecName "kube-api-access-4thmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:25.654268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.654216 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thmw\" (UniqueName: \"kubernetes.io/projected/6e2a5457-a145-4275-a212-af8b51930056-kube-api-access-4thmw\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:25.654268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.654239 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/9af75b68-2c68-4634-984a-51e174aa604a-kube-api-access-p4xqk\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:25.654268 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.654257 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a5457-a145-4275-a212-af8b51930056-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:25.788989 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.788961 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f"] Oct 30 01:35:25.803442 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.803416 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-364d04-6d7f96nt8f"] Oct 30 01:35:25.825834 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.825805 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596"] Oct 30 01:35:25.830091 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:25.830064 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-364d04-d4c64b96652596"] Oct 30 01:35:27.189568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:27.189507 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2a5457-a145-4275-a212-af8b51930056" path="/var/lib/kubelet/pods/6e2a5457-a145-4275-a212-af8b51930056/volumes" Oct 30 01:35:27.189957 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:27.189896 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af75b68-2c68-4634-984a-51e174aa604a" path="/var/lib/kubelet/pods/9af75b68-2c68-4634-984a-51e174aa604a/volumes" Oct 30 01:35:31.026576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:31.026543 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf"] Oct 30 01:35:31.028735 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:31.028710 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qsnkf"] Oct 30 01:35:31.190781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:31.190742 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543ea719-e826-47dd-a0de-3782ca439256" path="/var/lib/kubelet/pods/543ea719-e826-47dd-a0de-3782ca439256/volumes" Oct 30 01:35:34.226292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.226246 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48"] Oct 30 01:35:34.226828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.226533 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" podUID="7ef33b08-200a-4979-ac2e-856e3c89db4c" containerName="manager" containerID="cri-o://79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e" gracePeriod=10 Oct 30 01:35:34.494641 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.494616 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 01:35:34.500870 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.500845 4905 generic.go:356] "Generic (PLEG): container finished" podID="7ef33b08-200a-4979-ac2e-856e3c89db4c" containerID="79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e" exitCode=0 Oct 30 01:35:34.501014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.500885 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" Oct 30 01:35:34.501014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.500926 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" event={"ID":"7ef33b08-200a-4979-ac2e-856e3c89db4c","Type":"ContainerDied","Data":"79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e"} Oct 30 01:35:34.501014 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.500984 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48" event={"ID":"7ef33b08-200a-4979-ac2e-856e3c89db4c","Type":"ContainerDied","Data":"e8175c3746b3c90a624d0863d697ad57c45e0fb501f0d7ebefe810c11497bf9b"} Oct 30 01:35:34.501188 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.501014 4905 scope.go:122] "RemoveContainer" containerID="79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e" Oct 30 01:35:34.511097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.511079 4905 scope.go:122] "RemoveContainer" containerID="79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e" Oct 30 01:35:34.511355 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:35:34.511338 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e\": container with ID starting with 79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e not found: ID does not exist" containerID="79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e" Oct 30 01:35:34.511410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.511366 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e"} err="failed to get container status \"79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e\": rpc error: code = NotFound desc = could not find container \"79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e\": container with ID starting with 79372df7c8b6745a1cfaced2dc600685e245daf8f5777b83665b0a6aae1a5c1e not found: ID does not exist" Oct 30 01:35:34.636086 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.636056 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef33b08-200a-4979-ac2e-856e3c89db4c-kube-api-access-t4xll\" (UniqueName: \"kubernetes.io/projected/7ef33b08-200a-4979-ac2e-856e3c89db4c-kube-api-access-t4xll\") pod \"7ef33b08-200a-4979-ac2e-856e3c89db4c\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " Oct 30 01:35:34.636257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.636093 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef33b08-200a-4979-ac2e-856e3c89db4c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef33b08-200a-4979-ac2e-856e3c89db4c-extensions-socket-volume\") pod \"7ef33b08-200a-4979-ac2e-856e3c89db4c\" (UID: \"7ef33b08-200a-4979-ac2e-856e3c89db4c\") " Oct 30 01:35:34.636534 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.636482 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef33b08-200a-4979-ac2e-856e3c89db4c-extensions-socket-volume" pod "7ef33b08-200a-4979-ac2e-856e3c89db4c" (UID: "7ef33b08-200a-4979-ac2e-856e3c89db4c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:34.638203 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.638180 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef33b08-200a-4979-ac2e-856e3c89db4c-kube-api-access-t4xll" pod "7ef33b08-200a-4979-ac2e-856e3c89db4c" (UID: "7ef33b08-200a-4979-ac2e-856e3c89db4c"). InnerVolumeSpecName "kube-api-access-t4xll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:34.737812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.737782 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4xll\" (UniqueName: \"kubernetes.io/projected/7ef33b08-200a-4979-ac2e-856e3c89db4c-kube-api-access-t4xll\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:34.737812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.737810 4905 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ef33b08-200a-4979-ac2e-856e3c89db4c-extensions-socket-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:35:34.848411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.848377 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48"] Oct 30 01:35:34.862220 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:34.862192 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-drc48"] Oct 30 01:35:35.189533 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:35.189440 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef33b08-200a-4979-ac2e-856e3c89db4c" path="/var/lib/kubelet/pods/7ef33b08-200a-4979-ac2e-856e3c89db4c/volumes" Oct 30 01:35:42.026839 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:42.026800 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz"] Oct 30 01:35:42.040052 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:42.040023 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9kcwz"] Oct 30 01:35:42.413326 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:35:42.413250 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:35:43.191105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:43.191063 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d68dc9-d11a-425e-9a8d-3fca41ed8bb1" path="/var/lib/kubelet/pods/43d68dc9-d11a-425e-9a8d-3fca41ed8bb1/volumes" Oct 30 01:35:47.217890 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:35:47.217854 4905 scope.go:122] "RemoveContainer" containerID="7701a3d30d5ef2fee0dbe69378e6d068bc3f6ea3900cbea2f004ad3800910bec" Oct 30 01:36:10.027108 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:10.027068 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7"] Oct 30 01:36:10.040321 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:10.040283 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7kpv7"] Oct 30 01:36:11.189528 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:11.189483 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a634fd1-d117-4287-a406-44db7630535f" path="/var/lib/kubelet/pods/3a634fd1-d117-4287-a406-44db7630535f/volumes" Oct 30 01:36:12.413346 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:36:12.413314 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:36:21.035639 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:21.035548 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph"] Oct 30 01:36:21.048744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:21.048716 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bf5ph"] Oct 30 01:36:21.190279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:21.190249 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c65ce8-b406-489d-b51a-87c36444cbe5" path="/var/lib/kubelet/pods/f6c65ce8-b406-489d-b51a-87c36444cbe5/volumes" Oct 30 01:36:35.193304 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193272 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk"] Oct 30 01:36:35.193904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193883 4905 cpu_manager.go:414] "containerMap: removing container" podUID="9af75b68-2c68-4634-984a-51e174aa604a" containerName="payload-pre-processing" Oct 30 01:36:35.193998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193906 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af75b68-2c68-4634-984a-51e174aa604a" containerName="payload-pre-processing" Oct 30 01:36:35.193998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193939 4905 cpu_manager.go:414] "containerMap: removing container" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" Oct 30 01:36:35.193998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193947 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" Oct 30 01:36:35.193998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193960 4905 cpu_manager.go:414] "containerMap: removing container" podUID="4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" containerName="revoke-keys" Oct 30 01:36:35.193998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193967 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" containerName="revoke-keys" Oct 30 01:36:35.193998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193987 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6e2a5457-a145-4275-a212-af8b51930056" containerName="payload-processing" Oct 30 01:36:35.193998 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.193994 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2a5457-a145-4275-a212-af8b51930056" containerName="payload-processing" Oct 30 01:36:35.194386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.194026 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7ef33b08-200a-4979-ac2e-856e3c89db4c" containerName="manager" Oct 30 01:36:35.194386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.194034 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef33b08-200a-4979-ac2e-856e3c89db4c" containerName="manager" Oct 30 01:36:35.194386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.194140 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" Oct 30 01:36:35.194386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.194153 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" containerName="revoke-keys" Oct 30 01:36:35.194386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.194164 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2a5457-a145-4275-a212-af8b51930056" containerName="payload-processing" Oct 30 01:36:35.194386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.194178 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef33b08-200a-4979-ac2e-856e3c89db4c" containerName="manager" Oct 30 01:36:35.194386 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.194188 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af75b68-2c68-4634-984a-51e174aa604a" containerName="payload-pre-processing" Oct 30 01:36:35.198197 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.198175 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.214265 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.214236 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk"] Oct 30 01:36:35.225312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.225286 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cld2j\"" Oct 30 01:36:35.255402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.255376 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25adc5e3-09cf-4499-beef-1c6fe31e1706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5lhrk\" (UID: \"25adc5e3-09cf-4499-beef-1c6fe31e1706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.255556 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.255440 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/25adc5e3-09cf-4499-beef-1c6fe31e1706-kube-api-access-d8xm5\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5lhrk\" (UID: \"25adc5e3-09cf-4499-beef-1c6fe31e1706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.356402 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.356368 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25adc5e3-09cf-4499-beef-1c6fe31e1706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5lhrk\" (UID: \"25adc5e3-09cf-4499-beef-1c6fe31e1706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.356671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.356470 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/25adc5e3-09cf-4499-beef-1c6fe31e1706-kube-api-access-d8xm5\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5lhrk\" (UID: \"25adc5e3-09cf-4499-beef-1c6fe31e1706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.356826 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.356803 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25adc5e3-09cf-4499-beef-1c6fe31e1706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5lhrk\" (UID: \"25adc5e3-09cf-4499-beef-1c6fe31e1706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.381464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.381435 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/25adc5e3-09cf-4499-beef-1c6fe31e1706-kube-api-access-d8xm5\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5lhrk\" (UID: \"25adc5e3-09cf-4499-beef-1c6fe31e1706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.526949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.526919 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cld2j\"" Oct 30 01:36:35.532055 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.532034 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:35.855271 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:35.855244 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk"] Oct 30 01:36:35.855412 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:36:35.855388 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25adc5e3_09cf_4499_beef_1c6fe31e1706.slice/crio-aa4840419c0f02523deec0a8de57a69025030a6a5ec8eeba4cf51991e8b320e0 WatchSource:0}: Error finding container aa4840419c0f02523deec0a8de57a69025030a6a5ec8eeba4cf51991e8b320e0: Status 404 returned error can't find the container with id aa4840419c0f02523deec0a8de57a69025030a6a5ec8eeba4cf51991e8b320e0 Oct 30 01:36:36.830330 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:36.830291 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" event={"ID":"25adc5e3-09cf-4499-beef-1c6fe31e1706","Type":"ContainerStarted","Data":"b3011cb5a77c1861837521abf7aa346132c535e9dc45f0cb86ec2cc6c5060691"} Oct 30 01:36:36.830850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:36.830342 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" event={"ID":"25adc5e3-09cf-4499-beef-1c6fe31e1706","Type":"ContainerStarted","Data":"aa4840419c0f02523deec0a8de57a69025030a6a5ec8eeba4cf51991e8b320e0"} Oct 30 01:36:36.830850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:36.830495 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:36.905904 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:36.905856 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" podStartSLOduration=1.905841895 podStartE2EDuration="1.905841895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:36.890333381 +0000 UTC m=+3481.833506574" watchObservedRunningTime="2026-10-30 01:36:36.905841895 +0000 UTC m=+3481.849015089" Oct 30 01:36:42.027445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:42.027409 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb"] Oct 30 01:36:42.040929 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:42.040898 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xnltb"] Oct 30 01:36:42.414002 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:36:42.413915 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:36:43.189742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:43.189710 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d632d0-49d2-4273-93ed-b74df007430e" path="/var/lib/kubelet/pods/46d632d0-49d2-4273-93ed-b74df007430e/volumes" Oct 30 01:36:47.256275 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:47.256240 4905 scope.go:122] "RemoveContainer" containerID="f5c6bbfa08ed34ac8d8964af755c6e01d0378455645f44604c6a777ea4d2b496" Oct 30 01:36:47.265505 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:47.265484 4905 scope.go:122] "RemoveContainer" containerID="2df362b7828a9da9a5b4cf7d185c9ddd60739b822ff55fa4483ca97b432ccc71" Oct 30 01:36:47.838218 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:47.838188 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5lhrk" Oct 30 01:36:59.083780 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.083746 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq"] Oct 30 01:36:59.084267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.084194 4905 cpu_manager.go:414] "containerMap: removing container" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" Oct 30 01:36:59.084267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.084205 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" Oct 30 01:36:59.084349 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.084291 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae28e6e-e746-4576-92db-9210c7f39174" containerName="istio-proxy" Oct 30 01:36:59.088832 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.088815 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.108827 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.108798 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq"] Oct 30 01:36:59.119394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.119276 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:36:59.119394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.119353 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:36:59.119662 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.119409 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:36:59.119843 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.119826 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:36:59.240849 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.240811 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.241032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.240862 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnp9\" (UniqueName: \"kubernetes.io/projected/d8dc2195-abf2-4712-8913-1201fbdd9461-kube-api-access-qlnp9\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.241032 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.240993 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.342538 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.342431 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.342741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.342687 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.342818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.342745 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnp9\" (UniqueName: \"kubernetes.io/projected/d8dc2195-abf2-4712-8913-1201fbdd9461-kube-api-access-qlnp9\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.343573 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.343542 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.345121 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.345094 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.364080 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.364050 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnp9\" (UniqueName: \"kubernetes.io/projected/d8dc2195-abf2-4712-8913-1201fbdd9461-kube-api-access-qlnp9\") pod \"maas-api-revoke-sub-d576e68b-djqqq\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.410615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.410585 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:36:59.708649 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:36:59.708617 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dc2195_abf2_4712_8913_1201fbdd9461.slice/crio-e811f4fe42d2508f63a965b77e9dc85dac4bada34d9cb932b27e7b63158a296d WatchSource:0}: Error finding container e811f4fe42d2508f63a965b77e9dc85dac4bada34d9cb932b27e7b63158a296d: Status 404 returned error can't find the container with id e811f4fe42d2508f63a965b77e9dc85dac4bada34d9cb932b27e7b63158a296d Oct 30 01:36:59.708785 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.708698 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq"] Oct 30 01:36:59.971807 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:36:59.971735 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" event={"ID":"d8dc2195-abf2-4712-8913-1201fbdd9461","Type":"ContainerStarted","Data":"e811f4fe42d2508f63a965b77e9dc85dac4bada34d9cb932b27e7b63158a296d"} Oct 30 01:37:00.978624 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:00.978508 4905 generic.go:356] "Generic (PLEG): container finished" podID="d8dc2195-abf2-4712-8913-1201fbdd9461" containerID="2af22784009f7baa6b277b081f9d8c222051143ac55ef99ff89b8a7e37a238e4" exitCode=0 Oct 30 01:37:00.979005 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:00.978622 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" event={"ID":"d8dc2195-abf2-4712-8913-1201fbdd9461","Type":"ContainerDied","Data":"2af22784009f7baa6b277b081f9d8c222051143ac55ef99ff89b8a7e37a238e4"} Oct 30 01:37:02.131648 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.131624 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:37:02.273506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.273423 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-xks-ca\") pod \"d8dc2195-abf2-4712-8913-1201fbdd9461\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " Oct 30 01:37:02.273506 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.273467 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-openshift-ca\") pod \"d8dc2195-abf2-4712-8913-1201fbdd9461\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " Oct 30 01:37:02.273737 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.273565 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8dc2195-abf2-4712-8913-1201fbdd9461-kube-api-access-qlnp9\" (UniqueName: \"kubernetes.io/projected/d8dc2195-abf2-4712-8913-1201fbdd9461-kube-api-access-qlnp9\") pod \"d8dc2195-abf2-4712-8913-1201fbdd9461\" (UID: \"d8dc2195-abf2-4712-8913-1201fbdd9461\") " Oct 30 01:37:02.273895 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.273863 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-openshift-ca" pod "d8dc2195-abf2-4712-8913-1201fbdd9461" (UID: "d8dc2195-abf2-4712-8913-1201fbdd9461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:02.275804 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.275783 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-xks-ca" pod "d8dc2195-abf2-4712-8913-1201fbdd9461" (UID: "d8dc2195-abf2-4712-8913-1201fbdd9461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:02.275889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.275859 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dc2195-abf2-4712-8913-1201fbdd9461-kube-api-access-qlnp9" pod "d8dc2195-abf2-4712-8913-1201fbdd9461" (UID: "d8dc2195-abf2-4712-8913-1201fbdd9461"). InnerVolumeSpecName "kube-api-access-qlnp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:02.375012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.374981 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlnp9\" (UniqueName: \"kubernetes.io/projected/d8dc2195-abf2-4712-8913-1201fbdd9461-kube-api-access-qlnp9\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:37:02.375012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.375010 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:37:02.375012 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.375019 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc2195-abf2-4712-8913-1201fbdd9461-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:37:02.990954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.990916 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" event={"ID":"d8dc2195-abf2-4712-8913-1201fbdd9461","Type":"ContainerDied","Data":"e811f4fe42d2508f63a965b77e9dc85dac4bada34d9cb932b27e7b63158a296d"} Oct 30 01:37:02.990954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.990945 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq" Oct 30 01:37:02.991171 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:02.990969 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e811f4fe42d2508f63a965b77e9dc85dac4bada34d9cb932b27e7b63158a296d" Oct 30 01:37:03.026969 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:03.026938 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj"] Oct 30 01:37:03.029101 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:03.029074 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2lxpj"] Oct 30 01:37:03.190690 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:03.190656 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89bc660-a9ed-4e1c-81fe-9704b81a0dd4" path="/var/lib/kubelet/pods/e89bc660-a9ed-4e1c-81fe-9704b81a0dd4/volumes" Oct 30 01:37:12.415007 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:37:12.414971 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:37:15.583544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.583348 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9"] Oct 30 01:37:15.588541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.584121 4905 cpu_manager.go:414] "containerMap: removing container" podUID="d8dc2195-abf2-4712-8913-1201fbdd9461" containerName="revoke-keys" Oct 30 01:37:15.588541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.584145 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dc2195-abf2-4712-8913-1201fbdd9461" containerName="revoke-keys" Oct 30 01:37:15.588541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.584257 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dc2195-abf2-4712-8913-1201fbdd9461" containerName="revoke-keys" Oct 30 01:37:15.599431 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.598135 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.605350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.605317 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9"] Oct 30 01:37:15.615033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.615012 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:37:15.662837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.661640 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2rs7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:15.662837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.661728 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2rs7m" podUID="2e721927-370e-4f47-9eef-60330ef5886a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:15.676749 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.676697 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:37:15.677783 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.677151 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:37:15.677977 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.677725 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:37:15.773407 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.773363 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.773656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.773492 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.773656 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.773586 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqnh\" (UniqueName: \"kubernetes.io/projected/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-kube-api-access-7hqnh\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.847736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.847630 4905 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pwsj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:15.847736 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.847698 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pwsj2" podUID="e5fe03d5-2065-4957-a660-22bd064b5892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:15.874328 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.874293 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.874513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.874342 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.874513 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.874382 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqnh\" (UniqueName: \"kubernetes.io/projected/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-kube-api-access-7hqnh\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.875027 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.875007 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.876861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.876841 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.895238 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.895213 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqnh\" (UniqueName: \"kubernetes.io/projected/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-kube-api-access-7hqnh\") pod \"maas-api-revoke-sub-d2bc93bc-fbhc9\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:15.922565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:15.922541 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:16.215356 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:37:16.215324 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce01099e_9bd9_4c60_ab15_7bf1cd6a59d6.slice/crio-02eb49911b2e6614f16e1a85ac4bf9bf636da8b284e4921eeddc4f969b933dfd WatchSource:0}: Error finding container 02eb49911b2e6614f16e1a85ac4bf9bf636da8b284e4921eeddc4f969b933dfd: Status 404 returned error can't find the container with id 02eb49911b2e6614f16e1a85ac4bf9bf636da8b284e4921eeddc4f969b933dfd Oct 30 01:37:16.215661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:16.215640 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9"] Oct 30 01:37:17.086852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:17.086821 4905 generic.go:356] "Generic (PLEG): container finished" podID="ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" containerID="841c94d7b5463570cd47d2afcef01952879d45edcfd82b324ad7955502afe5ce" exitCode=0 Oct 30 01:37:17.087230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:17.086868 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" event={"ID":"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6","Type":"ContainerDied","Data":"841c94d7b5463570cd47d2afcef01952879d45edcfd82b324ad7955502afe5ce"} Oct 30 01:37:17.087230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:17.086930 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" event={"ID":"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6","Type":"ContainerStarted","Data":"02eb49911b2e6614f16e1a85ac4bf9bf636da8b284e4921eeddc4f969b933dfd"} Oct 30 01:37:18.239607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.239584 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:18.297537 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.297484 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-openshift-ca\") pod \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " Oct 30 01:37:18.297722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.297555 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-xks-ca\") pod \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " Oct 30 01:37:18.297722 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.297649 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-kube-api-access-7hqnh\" (UniqueName: \"kubernetes.io/projected/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-kube-api-access-7hqnh\") pod \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\" (UID: \"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6\") " Oct 30 01:37:18.297926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.297903 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-openshift-ca" pod "ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" (UID: "ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:18.298177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.298156 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:37:18.299810 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.299778 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-xks-ca" pod "ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" (UID: "ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:18.299918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.299824 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-kube-api-access-7hqnh" pod "ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" (UID: "ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6"). InnerVolumeSpecName "kube-api-access-7hqnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:18.399033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.398937 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqnh\" (UniqueName: \"kubernetes.io/projected/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-kube-api-access-7hqnh\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:37:18.399033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:18.398982 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.099985 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:19.099958 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" Oct 30 01:37:19.100152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:19.099952 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9" event={"ID":"ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6","Type":"ContainerDied","Data":"02eb49911b2e6614f16e1a85ac4bf9bf636da8b284e4921eeddc4f969b933dfd"} Oct 30 01:37:19.100152 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:19.100087 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02eb49911b2e6614f16e1a85ac4bf9bf636da8b284e4921eeddc4f969b933dfd" Oct 30 01:37:23.026889 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:23.026852 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm"] Oct 30 01:37:23.040474 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:23.040437 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-k5qqm"] Oct 30 01:37:23.189246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:23.189211 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623af2c3-1233-46fb-9e46-76acaf0c3104" path="/var/lib/kubelet/pods/623af2c3-1233-46fb-9e46-76acaf0c3104/volumes" Oct 30 01:37:27.612946 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.612891 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w"] Oct 30 01:37:27.614400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.614355 4905 cpu_manager.go:414] "containerMap: removing container" podUID="ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" containerName="revoke-keys" Oct 30 01:37:27.614614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.614594 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" containerName="revoke-keys" Oct 30 01:37:27.614894 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.614867 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" containerName="revoke-keys" Oct 30 01:37:27.619123 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.619101 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.635682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.635655 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-646191-openshift-default-dockercfg-757hr\"" Oct 30 01:37:27.653560 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.652988 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w"] Oct 30 01:37:27.691125 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691092 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-podinfo\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691147 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl45n\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-kube-api-access-rl45n\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691290 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691258 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-socket\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691355 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0db35f5-5f3a-4d01-b871-c14e81b61471-istiod-ca-cert\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691381 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-envoy\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691421 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691398 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-data\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691576 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-certs\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691655 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691636 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-credential-socket\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.691725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.691686 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-token\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792396 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0db35f5-5f3a-4d01-b871-c14e81b61471-istiod-ca-cert\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792462 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-envoy\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792488 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-data\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792586 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-certs\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792627 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-credential-socket\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792652 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-token\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792762 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-podinfo\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792808 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl45n\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-kube-api-access-rl45n\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.792846 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-socket\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.793336 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-socket\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.794023 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0db35f5-5f3a-4d01-b871-c14e81b61471-istiod-ca-cert\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.794618 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-credential-socket\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.794875 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-data\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.800549 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.795085 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-certs\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.823542 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.822052 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-envoy\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.842354 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.842310 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl45n\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-kube-api-access-rl45n\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.843167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.843142 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-token\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.843306 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.843261 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-podinfo\") pod \"e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:27.942266 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:27.942178 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:28.253471 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:28.253443 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w"] Oct 30 01:37:28.253793 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:37:28.253769 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0db35f5_5f3a_4d01_b871_c14e81b61471.slice/crio-1c077582a54fbdf1d649a7a0f13fa675214b41404ba251c7eb39c0df98d85abc WatchSource:0}: Error finding container 1c077582a54fbdf1d649a7a0f13fa675214b41404ba251c7eb39c0df98d85abc: Status 404 returned error can't find the container with id 1c077582a54fbdf1d649a7a0f13fa675214b41404ba251c7eb39c0df98d85abc Oct 30 01:37:28.255989 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:28.255953 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:37:28.256082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:28.256027 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:37:28.256082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:28.256056 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:37:29.155609 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:29.155568 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" event={"ID":"b0db35f5-5f3a-4d01-b871-c14e81b61471","Type":"ContainerStarted","Data":"bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c"} Oct 30 01:37:29.156031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:29.155623 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" event={"ID":"b0db35f5-5f3a-4d01-b871-c14e81b61471","Type":"ContainerStarted","Data":"1c077582a54fbdf1d649a7a0f13fa675214b41404ba251c7eb39c0df98d85abc"} Oct 30 01:37:29.243069 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:29.243017 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podStartSLOduration=2.243004208 podStartE2EDuration="2.243004208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:29.240283243 +0000 UTC m=+3534.183456436" watchObservedRunningTime="2026-10-30 01:37:29.243004208 +0000 UTC m=+3534.186177394" Oct 30 01:37:29.942582 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:29.942541 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:29.947564 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:29.947511 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:30.162936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:30.162901 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:30.164022 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:30.163997 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:33.414043 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.414009 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl"] Oct 30 01:37:33.418144 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.418120 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.447813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.447766 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-646191\"" Oct 30 01:37:33.447813 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.447779 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0ff044e4-dockercfg-v5h4l\"" Oct 30 01:37:33.450962 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.450938 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl"] Oct 30 01:37:33.514764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.512245 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w"] Oct 30 01:37:33.520115 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.520086 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:33.536110 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.536081 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w"] Oct 30 01:37:33.553631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.553596 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmxs\" (UniqueName: \"kubernetes.io/projected/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-kube-api-access-xkmxs\") pod \"payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.553794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.553727 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.654719 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.654681 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.654905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.654771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:33.654905 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.654816 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7csl\" (UniqueName: \"kubernetes.io/projected/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-kube-api-access-v7csl\") pod \"payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:33.655038 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.654953 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmxs\" (UniqueName: \"kubernetes.io/projected/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-kube-api-access-xkmxs\") pod \"payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.655399 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.655379 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.675645 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.675576 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmxs\" (UniqueName: \"kubernetes.io/projected/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-kube-api-access-xkmxs\") pod \"payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.731370 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.731335 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:33.755938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.755910 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:33.756097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.755957 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7csl\" (UniqueName: \"kubernetes.io/projected/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-kube-api-access-v7csl\") pod \"payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:33.756646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.756578 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:33.776781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.776733 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7csl\" (UniqueName: \"kubernetes.io/projected/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-kube-api-access-v7csl\") pod \"payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:33.838172 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:33.838124 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:34.027860 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:34.027829 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl"] Oct 30 01:37:34.027860 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:37:34.027845 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae0ff6e_d242_4abe_95fa_c00abd0cd0d1.slice/crio-10fc86b1bca3bf7fbd9df04310c33346077c5f03c91889d048516ee36bc512bc WatchSource:0}: Error finding container 10fc86b1bca3bf7fbd9df04310c33346077c5f03c91889d048516ee36bc512bc: Status 404 returned error can't find the container with id 10fc86b1bca3bf7fbd9df04310c33346077c5f03c91889d048516ee36bc512bc Oct 30 01:37:34.149986 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:34.149957 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w"] Oct 30 01:37:34.151968 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:37:34.151944 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8841d99e_3c99_4e8e_90d3_6f695a91bcb8.slice/crio-a4964434630519174852b0140be772260496eda05b69a0f61f47435fa14f402b WatchSource:0}: Error finding container a4964434630519174852b0140be772260496eda05b69a0f61f47435fa14f402b: Status 404 returned error can't find the container with id a4964434630519174852b0140be772260496eda05b69a0f61f47435fa14f402b Oct 30 01:37:34.189444 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:34.189410 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" event={"ID":"8841d99e-3c99-4e8e-90d3-6f695a91bcb8","Type":"ContainerStarted","Data":"a4964434630519174852b0140be772260496eda05b69a0f61f47435fa14f402b"} Oct 30 01:37:34.191571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:34.191545 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" event={"ID":"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1","Type":"ContainerStarted","Data":"834bc7e4db28973cf91476ed02b00d9eaec8dd969c7631e41595b963089c5a2b"} Oct 30 01:37:34.191688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:34.191588 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" event={"ID":"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1","Type":"ContainerStarted","Data":"10fc86b1bca3bf7fbd9df04310c33346077c5f03c91889d048516ee36bc512bc"} Oct 30 01:37:34.191763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:34.191697 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:34.231679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:34.231632 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" podStartSLOduration=1.231618206 podStartE2EDuration="1.231618206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:34.231008013 +0000 UTC m=+3539.174181206" watchObservedRunningTime="2026-10-30 01:37:34.231618206 +0000 UTC m=+3539.174791396" Oct 30 01:37:35.198814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:35.198768 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" event={"ID":"8841d99e-3c99-4e8e-90d3-6f695a91bcb8","Type":"ContainerStarted","Data":"db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7"} Oct 30 01:37:35.199329 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:35.198961 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:35.241633 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:35.241574 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" podStartSLOduration=2.241555891 podStartE2EDuration="2.241555891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:35.240641484 +0000 UTC m=+3540.183814677" watchObservedRunningTime="2026-10-30 01:37:35.241555891 +0000 UTC m=+3540.184729082" Oct 30 01:37:41.043547 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:41.035678 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk"] Oct 30 01:37:41.053545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:41.051169 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d5pxk"] Oct 30 01:37:41.193230 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:41.193191 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5992f34e-bc76-4641-973f-7461355cd585" path="/var/lib/kubelet/pods/5992f34e-bc76-4641-973f-7461355cd585/volumes" Oct 30 01:37:42.414743 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:37:42.414700 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:37:45.202358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:45.202277 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:37:46.025671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.024665 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": read tcp 10.132.0.2:57868->10.132.1.99:15021: read: connection reset by peer" start-of-body= Oct 30 01:37:46.025671 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.024722 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": read tcp 10.132.0.2:57868->10.132.1.99:15021: read: connection reset by peer" Oct 30 01:37:46.210267 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.210196 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:37:46.298970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.298876 4905 generic.go:356] "Generic (PLEG): container finished" podID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerID="bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c" exitCode=0 Oct 30 01:37:46.298970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.298946 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" event={"ID":"b0db35f5-5f3a-4d01-b871-c14e81b61471","Type":"ContainerDied","Data":"bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c"} Oct 30 01:37:46.299343 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.299326 4905 scope.go:122] "RemoveContainer" containerID="bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c" Oct 30 01:37:46.300703 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.300671 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:37:46.300795 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.300760 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:37:46.300836 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:46.300803 4905 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 01:37:47.290947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:47.290911 4905 scope.go:122] "RemoveContainer" containerID="cf6233c3f97f87422ed90905620bd7be60400f1b1113a5e99c74fc98c9fa1db9" Oct 30 01:37:47.302647 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:47.302334 4905 scope.go:122] "RemoveContainer" containerID="e270addc52cab35c394401c3675b9aeeb6019ecd926779e46dcd1d0dd2342c1a" Oct 30 01:37:47.308947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:47.308922 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" event={"ID":"b0db35f5-5f3a-4d01-b871-c14e81b61471","Type":"ContainerStarted","Data":"d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17"} Oct 30 01:37:47.311646 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:47.311631 4905 scope.go:122] "RemoveContainer" containerID="d13b6d6e4260de656a32dad4676b2eb3737fd41e59be96c0a7f4ca3e725850d4" Oct 30 01:37:47.944544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:47.943929 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:47.944544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:47.944103 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 30 01:37:47.944544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:47.944157 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 30 01:37:48.950809 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:48.950767 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 30 01:37:48.951242 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:48.950831 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 30 01:37:49.943042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:49.943005 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 30 01:37:49.943416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:49.943373 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 30 01:37:50.946543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:50.945918 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 30 01:37:50.946543 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:50.945986 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 30 01:37:51.945545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:51.943880 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 30 01:37:51.945545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:51.943954 4905 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 30 01:37:52.946569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:52.946513 4905 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:52.947106 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:52.946949 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:37:52.947873 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:37:52.947856 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:38:08.021924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.021889 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st"] Oct 30 01:38:08.026742 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.026715 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.046568 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.046542 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:08.062630 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.062600 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st"] Oct 30 01:38:08.062788 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.062746 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:08.062871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.062836 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:38:08.062936 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.062906 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:08.088490 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.088458 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.088638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.088546 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvnp\" (UniqueName: \"kubernetes.io/projected/fbd6957f-30b3-473d-b6cc-38eef899c898-kube-api-access-2gvnp\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.088704 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.088680 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.189831 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.189797 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.190004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.189857 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvnp\" (UniqueName: \"kubernetes.io/projected/fbd6957f-30b3-473d-b6cc-38eef899c898-kube-api-access-2gvnp\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.190004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.189913 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.190464 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.190442 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.192695 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.192672 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.213589 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.213565 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvnp\" (UniqueName: \"kubernetes.io/projected/fbd6957f-30b3-473d-b6cc-38eef899c898-kube-api-access-2gvnp\") pod \"maas-api-revoke-sub-36288509-9h5st\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.341050 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.340956 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:08.638900 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:38:08.638870 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd6957f_30b3_473d_b6cc_38eef899c898.slice/crio-c54ced733d4c4516cce51197f3a6a74c31181e68550067a7e03df413b2599196 WatchSource:0}: Error finding container c54ced733d4c4516cce51197f3a6a74c31181e68550067a7e03df413b2599196: Status 404 returned error can't find the container with id c54ced733d4c4516cce51197f3a6a74c31181e68550067a7e03df413b2599196 Oct 30 01:38:08.640352 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:08.640306 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st"] Oct 30 01:38:09.462412 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:09.462360 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" event={"ID":"fbd6957f-30b3-473d-b6cc-38eef899c898","Type":"ContainerStarted","Data":"637bbf9e203d364a45a0c368ec9e287d753642915949635f4b06e00710005267"} Oct 30 01:38:09.462828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:09.462431 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" event={"ID":"fbd6957f-30b3-473d-b6cc-38eef899c898","Type":"ContainerStarted","Data":"c54ced733d4c4516cce51197f3a6a74c31181e68550067a7e03df413b2599196"} Oct 30 01:38:09.505597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:09.505550 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" podStartSLOduration=2.505536325 podStartE2EDuration="2.505536325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:09.502836066 +0000 UTC m=+3574.446009259" watchObservedRunningTime="2026-10-30 01:38:09.505536325 +0000 UTC m=+3574.448709518" Oct 30 01:38:10.470272 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:10.470236 4905 generic.go:356] "Generic (PLEG): container finished" podID="fbd6957f-30b3-473d-b6cc-38eef899c898" containerID="637bbf9e203d364a45a0c368ec9e287d753642915949635f4b06e00710005267" exitCode=0 Oct 30 01:38:10.470765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:10.470306 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" event={"ID":"fbd6957f-30b3-473d-b6cc-38eef899c898","Type":"ContainerDied","Data":"637bbf9e203d364a45a0c368ec9e287d753642915949635f4b06e00710005267"} Oct 30 01:38:11.623493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.623471 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:11.729033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.728955 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-openshift-ca\") pod \"fbd6957f-30b3-473d-b6cc-38eef899c898\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " Oct 30 01:38:11.729217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.729062 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd6957f-30b3-473d-b6cc-38eef899c898-kube-api-access-2gvnp\" (UniqueName: \"kubernetes.io/projected/fbd6957f-30b3-473d-b6cc-38eef899c898-kube-api-access-2gvnp\") pod \"fbd6957f-30b3-473d-b6cc-38eef899c898\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " Oct 30 01:38:11.729217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.729128 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-xks-ca\") pod \"fbd6957f-30b3-473d-b6cc-38eef899c898\" (UID: \"fbd6957f-30b3-473d-b6cc-38eef899c898\") " Oct 30 01:38:11.729392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.729368 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-openshift-ca" pod "fbd6957f-30b3-473d-b6cc-38eef899c898" (UID: "fbd6957f-30b3-473d-b6cc-38eef899c898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:11.731360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.731326 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-xks-ca" pod "fbd6957f-30b3-473d-b6cc-38eef899c898" (UID: "fbd6957f-30b3-473d-b6cc-38eef899c898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:11.731360 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.731346 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd6957f-30b3-473d-b6cc-38eef899c898-kube-api-access-2gvnp" pod "fbd6957f-30b3-473d-b6cc-38eef899c898" (UID: "fbd6957f-30b3-473d-b6cc-38eef899c898"). InnerVolumeSpecName "kube-api-access-2gvnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:11.830090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.830059 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvnp\" (UniqueName: \"kubernetes.io/projected/fbd6957f-30b3-473d-b6cc-38eef899c898-kube-api-access-2gvnp\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:11.830090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.830087 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:11.830280 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:11.830102 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd6957f-30b3-473d-b6cc-38eef899c898-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:12.413872 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:38:12.413831 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:38:12.482893 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:12.482868 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" Oct 30 01:38:12.483065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:12.482912 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st" event={"ID":"fbd6957f-30b3-473d-b6cc-38eef899c898","Type":"ContainerDied","Data":"c54ced733d4c4516cce51197f3a6a74c31181e68550067a7e03df413b2599196"} Oct 30 01:38:12.483065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:12.482955 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c54ced733d4c4516cce51197f3a6a74c31181e68550067a7e03df413b2599196" Oct 30 01:38:19.046594 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.046560 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6"] Oct 30 01:38:19.047118 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.047101 4905 cpu_manager.go:414] "containerMap: removing container" podUID="fbd6957f-30b3-473d-b6cc-38eef899c898" containerName="revoke-keys" Oct 30 01:38:19.047167 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.047120 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd6957f-30b3-473d-b6cc-38eef899c898" containerName="revoke-keys" Oct 30 01:38:19.047202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.047192 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd6957f-30b3-473d-b6cc-38eef899c898" containerName="revoke-keys" Oct 30 01:38:19.050498 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.050478 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.088562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.081257 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:19.088562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.081313 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:19.088562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.081328 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:19.088562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.081600 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:38:19.088562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.083943 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6"] Oct 30 01:38:19.205453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.205417 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/3b12d0d9-345e-4d7d-a874-3adad424a75e-kube-api-access-6fcfs\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.205665 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.205581 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.205724 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.205693 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.307000 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.306918 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.307000 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.306970 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.307255 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.307060 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/3b12d0d9-345e-4d7d-a874-3adad424a75e-kube-api-access-6fcfs\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.307685 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.307666 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.309406 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.309385 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.327954 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.327910 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/3b12d0d9-345e-4d7d-a874-3adad424a75e-kube-api-access-6fcfs\") pod \"maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.364453 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.364425 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:19.663084 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:19.663051 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6"] Oct 30 01:38:19.663237 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:38:19.663155 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b12d0d9_345e_4d7d_a874_3adad424a75e.slice/crio-76a93b9dfff96f8b52e4f928ca7b9322cf6284f5ca8e6e519a011a780ef10ab2 WatchSource:0}: Error finding container 76a93b9dfff96f8b52e4f928ca7b9322cf6284f5ca8e6e519a011a780ef10ab2: Status 404 returned error can't find the container with id 76a93b9dfff96f8b52e4f928ca7b9322cf6284f5ca8e6e519a011a780ef10ab2 Oct 30 01:38:20.528279 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:20.528242 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" event={"ID":"3b12d0d9-345e-4d7d-a874-3adad424a75e","Type":"ContainerStarted","Data":"21b40b236358199634b98490d783f0ab27f3dd859b8cf1c4deba24c0de9510ae"} Oct 30 01:38:20.528764 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:20.528300 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" event={"ID":"3b12d0d9-345e-4d7d-a874-3adad424a75e","Type":"ContainerStarted","Data":"76a93b9dfff96f8b52e4f928ca7b9322cf6284f5ca8e6e519a011a780ef10ab2"} Oct 30 01:38:20.570170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:20.570128 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" podStartSLOduration=1.570115586 podStartE2EDuration="1.570115586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:20.568531208 +0000 UTC m=+3585.511704397" watchObservedRunningTime="2026-10-30 01:38:20.570115586 +0000 UTC m=+3585.513288778" Oct 30 01:38:21.534978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:21.534943 4905 generic.go:356] "Generic (PLEG): container finished" podID="3b12d0d9-345e-4d7d-a874-3adad424a75e" containerID="21b40b236358199634b98490d783f0ab27f3dd859b8cf1c4deba24c0de9510ae" exitCode=0 Oct 30 01:38:21.535396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:21.535031 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" event={"ID":"3b12d0d9-345e-4d7d-a874-3adad424a75e","Type":"ContainerDied","Data":"21b40b236358199634b98490d783f0ab27f3dd859b8cf1c4deba24c0de9510ae"} Oct 30 01:38:22.697968 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.697934 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:22.847205 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.847112 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b12d0d9-345e-4d7d-a874-3adad424a75e-kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/3b12d0d9-345e-4d7d-a874-3adad424a75e-kube-api-access-6fcfs\") pod \"3b12d0d9-345e-4d7d-a874-3adad424a75e\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " Oct 30 01:38:22.847378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.847205 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-xks-ca\") pod \"3b12d0d9-345e-4d7d-a874-3adad424a75e\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " Oct 30 01:38:22.847378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.847236 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-openshift-ca\") pod \"3b12d0d9-345e-4d7d-a874-3adad424a75e\" (UID: \"3b12d0d9-345e-4d7d-a874-3adad424a75e\") " Oct 30 01:38:22.847694 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.847670 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-openshift-ca" pod "3b12d0d9-345e-4d7d-a874-3adad424a75e" (UID: "3b12d0d9-345e-4d7d-a874-3adad424a75e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:22.849371 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.849348 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b12d0d9-345e-4d7d-a874-3adad424a75e-kube-api-access-6fcfs" pod "3b12d0d9-345e-4d7d-a874-3adad424a75e" (UID: "3b12d0d9-345e-4d7d-a874-3adad424a75e"). InnerVolumeSpecName "kube-api-access-6fcfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:22.849754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.849737 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-xks-ca" pod "3b12d0d9-345e-4d7d-a874-3adad424a75e" (UID: "3b12d0d9-345e-4d7d-a874-3adad424a75e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:22.948186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.948149 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/3b12d0d9-345e-4d7d-a874-3adad424a75e-kube-api-access-6fcfs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:22.948186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.948179 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:22.948186 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:22.948188 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b12d0d9-345e-4d7d-a874-3adad424a75e-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:23.549410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:23.549369 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" event={"ID":"3b12d0d9-345e-4d7d-a874-3adad424a75e","Type":"ContainerDied","Data":"76a93b9dfff96f8b52e4f928ca7b9322cf6284f5ca8e6e519a011a780ef10ab2"} Oct 30 01:38:23.549658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:23.549430 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76a93b9dfff96f8b52e4f928ca7b9322cf6284f5ca8e6e519a011a780ef10ab2" Oct 30 01:38:23.549658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:23.549389 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6" Oct 30 01:38:29.498604 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:29.498570 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w"] Oct 30 01:38:29.532683 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:29.532655 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl"] Oct 30 01:38:31.108820 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:31.108779 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6"] Oct 30 01:38:31.124593 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:31.122997 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-646191-dczz6"] Oct 30 01:38:31.189344 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:31.189312 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b12d0d9-345e-4d7d-a874-3adad424a75e" path="/var/lib/kubelet/pods/3b12d0d9-345e-4d7d-a874-3adad424a75e/volumes" Oct 30 01:38:34.499460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:34.499386 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" podUID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" containerName="payload-processing" containerID="cri-o://db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7" gracePeriod=25 Oct 30 01:38:34.533854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:34.533815 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" podUID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" containerName="payload-pre-processing" containerID="cri-o://834bc7e4db28973cf91476ed02b00d9eaec8dd969c7631e41595b963089c5a2b" gracePeriod=25 Oct 30 01:38:35.200224 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:35.200189 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" podUID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:38:36.208181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:36.208143 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" podUID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:38:36.588504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:36.588473 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:38:36.590618 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:36.590596 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:38:37.918034 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:37.917985 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w"] Oct 30 01:38:37.918567 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:37.918270 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" containerID="cri-o://d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17" gracePeriod=30 Oct 30 01:38:37.949252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:37.949227 4905 patch_prober.go:29] interesting pod/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:38:37.949393 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:37.949280 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:38:42.435880 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:38:42.435830 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:38:43.187150 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.187096 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:38:43.282375 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282342 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-certs\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.282566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282403 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-kube-api-access-rl45n\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-kube-api-access-rl45n\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.282566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282458 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-podinfo\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.282566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282490 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-credential-socket\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.282763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282569 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-data\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.282763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282637 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-envoy\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.282763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282636 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-certs" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:43.282763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282667 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-token\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-token\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.282763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282709 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-socket\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.283033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282801 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0db35f5-5f3a-4d01-b871-c14e81b61471-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0db35f5-5f3a-4d01-b871-c14e81b61471-istiod-ca-cert\") pod \"b0db35f5-5f3a-4d01-b871-c14e81b61471\" (UID: \"b0db35f5-5f3a-4d01-b871-c14e81b61471\") " Oct 30 01:38:43.283033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282828 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-credential-socket" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:43.283033 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.282997 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-socket" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:43.283200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.283112 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-data" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:43.283200 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.283176 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0db35f5-5f3a-4d01-b871-c14e81b61471-istiod-ca-cert" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:43.283394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.283375 4905 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.283572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.283555 4905 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0db35f5-5f3a-4d01-b871-c14e81b61471-istiod-ca-cert\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.283725 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.283702 4905 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-workload-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.283844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.283729 4905 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-credential-socket\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.283844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.283744 4905 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-data\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.286550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.286009 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-podinfo" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:43.286550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.286092 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-envoy" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:43.286550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.286142 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-kube-api-access-rl45n" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "kube-api-access-rl45n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:43.286550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.286505 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-token" pod "b0db35f5-5f3a-4d01-b871-c14e81b61471" (UID: "b0db35f5-5f3a-4d01-b871-c14e81b61471"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:43.385337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.385301 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl45n\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-kube-api-access-rl45n\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.385337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.385341 4905 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-podinfo\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.385638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.385356 4905 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-envoy\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.385638 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.385369 4905 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0db35f5-5f3a-4d01-b871-c14e81b61471-istio-token\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:43.669100 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.669064 4905 generic.go:356] "Generic (PLEG): container finished" podID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerID="d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17" exitCode=0 Oct 30 01:38:43.669621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.669111 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" Oct 30 01:38:43.669621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.669143 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" event={"ID":"b0db35f5-5f3a-4d01-b871-c14e81b61471","Type":"ContainerDied","Data":"d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17"} Oct 30 01:38:43.669621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.669214 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w" event={"ID":"b0db35f5-5f3a-4d01-b871-c14e81b61471","Type":"ContainerDied","Data":"1c077582a54fbdf1d649a7a0f13fa675214b41404ba251c7eb39c0df98d85abc"} Oct 30 01:38:43.669621 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.669251 4905 scope.go:122] "RemoveContainer" containerID="d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17" Oct 30 01:38:43.679942 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.679924 4905 scope.go:122] "RemoveContainer" containerID="bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c" Oct 30 01:38:43.688658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.688641 4905 scope.go:122] "RemoveContainer" containerID="d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17" Oct 30 01:38:43.688956 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:38:43.688923 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17\": container with ID starting with d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17 not found: ID does not exist" containerID="d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17" Oct 30 01:38:43.688956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.688952 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17"} err="failed to get container status \"d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17\": rpc error: code = NotFound desc = could not find container \"d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17\": container with ID starting with d4e438b7baf002a93fe23a911e4b0f2d8e30edcde37ca1e5c8b562f8bee15b17 not found: ID does not exist" Oct 30 01:38:43.689097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.688969 4905 scope.go:122] "RemoveContainer" containerID="bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c" Oct 30 01:38:43.689234 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:38:43.689217 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c\": container with ID starting with bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c not found: ID does not exist" containerID="bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c" Oct 30 01:38:43.689292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.689242 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c"} err="failed to get container status \"bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c\": rpc error: code = NotFound desc = could not find container \"bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c\": container with ID starting with bbed710b3df0f28cc9ef31bf7f48f5000a95b2374a99036bac642b1b6fde5a5c not found: ID does not exist" Oct 30 01:38:43.725947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.725921 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w"] Oct 30 01:38:43.729426 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:43.729409 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-646191-openshift-default-77498c7fcc92p5w"] Oct 30 01:38:45.202289 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:45.202250 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" path="/var/lib/kubelet/pods/b0db35f5-5f3a-4d01-b871-c14e81b61471/volumes" Oct 30 01:38:45.205754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:45.205716 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" podUID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:38:46.221085 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:46.221027 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" podUID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:38:47.380883 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:47.380852 4905 scope.go:122] "RemoveContainer" containerID="e850713337e5ae2b8c436c31ea2853cffa1537680568f430b7fa2aac4c854f31" Oct 30 01:38:47.441054 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:47.441027 4905 scope.go:122] "RemoveContainer" containerID="c4e27045a8932391d2a70f72efe8240a010797da031f43719eb7311cd1b7b201" Oct 30 01:38:47.518554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:47.514678 4905 scope.go:122] "RemoveContainer" containerID="2af246774e4181028ec56f049d8aa40980371be4f7a1348014066b648fe64715" Oct 30 01:38:54.793534 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.793497 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:38:54.795661 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.795635 4905 generic.go:356] "Generic (PLEG): container finished" podID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" containerID="db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7" exitCode=0 Oct 30 01:38:54.795784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.795670 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" Oct 30 01:38:54.795784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.795713 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" event={"ID":"8841d99e-3c99-4e8e-90d3-6f695a91bcb8","Type":"ContainerDied","Data":"db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7"} Oct 30 01:38:54.795784 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.795775 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w" event={"ID":"8841d99e-3c99-4e8e-90d3-6f695a91bcb8","Type":"ContainerDied","Data":"a4964434630519174852b0140be772260496eda05b69a0f61f47435fa14f402b"} Oct 30 01:38:54.795960 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.795805 4905 scope.go:122] "RemoveContainer" containerID="db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7" Oct 30 01:38:54.797818 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.797796 4905 generic.go:356] "Generic (PLEG): container finished" podID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" containerID="834bc7e4db28973cf91476ed02b00d9eaec8dd969c7631e41595b963089c5a2b" exitCode=0 Oct 30 01:38:54.797923 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.797842 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" event={"ID":"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1","Type":"ContainerDied","Data":"834bc7e4db28973cf91476ed02b00d9eaec8dd969c7631e41595b963089c5a2b"} Oct 30 01:38:54.805162 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.805140 4905 scope.go:122] "RemoveContainer" containerID="db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7" Oct 30 01:38:54.805550 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:38:54.805446 4905 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7\": container with ID starting with db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7 not found: ID does not exist" containerID="db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7" Oct 30 01:38:54.805550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.805478 4905 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7"} err="failed to get container status \"db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7\": rpc error: code = NotFound desc = could not find container \"db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7\": container with ID starting with db4431802e93bbc4484c3392154850d971719c67cfb68c748d71f252abfff3f7 not found: ID does not exist" Oct 30 01:38:54.822363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.822332 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:38:54.865595 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.865573 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-kube-api-access-xkmxs\" (UniqueName: \"kubernetes.io/projected/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-kube-api-access-xkmxs\") pod \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " Oct 30 01:38:54.865744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.865624 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-kube-api-access-v7csl\" (UniqueName: \"kubernetes.io/projected/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-kube-api-access-v7csl\") pod \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " Oct 30 01:38:54.865744 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.865669 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-plugins-config-volume\") pod \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\" (UID: \"8841d99e-3c99-4e8e-90d3-6f695a91bcb8\") " Oct 30 01:38:54.865881 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.865770 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-plugins-config-volume\") pod \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\" (UID: \"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1\") " Oct 30 01:38:54.866207 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.866176 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-plugins-config-volume" pod "8841d99e-3c99-4e8e-90d3-6f695a91bcb8" (UID: "8841d99e-3c99-4e8e-90d3-6f695a91bcb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:54.866852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.866825 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-plugins-config-volume" pod "6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" (UID: "6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:54.867876 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.867852 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-kube-api-access-xkmxs" pod "6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" (UID: "6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1"). InnerVolumeSpecName "kube-api-access-xkmxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:54.868061 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.868036 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-kube-api-access-v7csl" pod "8841d99e-3c99-4e8e-90d3-6f695a91bcb8" (UID: "8841d99e-3c99-4e8e-90d3-6f695a91bcb8"). InnerVolumeSpecName "kube-api-access-v7csl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:54.967373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.967344 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:54.967373 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.967369 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmxs\" (UniqueName: \"kubernetes.io/projected/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1-kube-api-access-xkmxs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:54.967599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.967384 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7csl\" (UniqueName: \"kubernetes.io/projected/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-kube-api-access-v7csl\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:54.967599 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:54.967396 4905 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8841d99e-3c99-4e8e-90d3-6f695a91bcb8-plugins-config-volume\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:38:55.143469 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.143445 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w"] Oct 30 01:38:55.147044 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.147025 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-646191-64dcfc75bvlb4w"] Oct 30 01:38:55.189368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.189342 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" path="/var/lib/kubelet/pods/8841d99e-3c99-4e8e-90d3-6f695a91bcb8/volumes" Oct 30 01:38:55.804801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.804726 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" Oct 30 01:38:55.804801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.804738 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl" event={"ID":"6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1","Type":"ContainerDied","Data":"10fc86b1bca3bf7fbd9df04310c33346077c5f03c91889d048516ee36bc512bc"} Oct 30 01:38:55.804801 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.804801 4905 scope.go:122] "RemoveContainer" containerID="834bc7e4db28973cf91476ed02b00d9eaec8dd969c7631e41595b963089c5a2b" Oct 30 01:38:55.859105 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.859079 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl"] Oct 30 01:38:55.872761 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:55.872740 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-646191-7b4d76h9wl"] Oct 30 01:38:57.189287 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:38:57.189253 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" path="/var/lib/kubelet/pods/6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1/volumes" Oct 30 01:39:12.414314 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:39:12.414271 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:39:20.603608 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.603530 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p"] Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.603983 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.603996 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604009 4905 cpu_manager.go:414] "containerMap: removing container" podUID="3b12d0d9-345e-4d7d-a874-3adad424a75e" containerName="revoke-keys" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604014 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12d0d9-345e-4d7d-a874-3adad424a75e" containerName="revoke-keys" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604036 4905 cpu_manager.go:414] "containerMap: removing container" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604041 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604057 4905 cpu_manager.go:414] "containerMap: removing container" podUID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" containerName="payload-processing" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604061 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" containerName="payload-processing" Oct 30 01:39:20.604065 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604069 4905 cpu_manager.go:414] "containerMap: removing container" podUID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" containerName="payload-pre-processing" Oct 30 01:39:20.604363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604074 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" containerName="payload-pre-processing" Oct 30 01:39:20.604363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604138 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="8841d99e-3c99-4e8e-90d3-6f695a91bcb8" containerName="payload-processing" Oct 30 01:39:20.604363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604161 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae0ff6e-d242-4abe-95fa-c00abd0cd0d1" containerName="payload-pre-processing" Oct 30 01:39:20.604363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604169 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" Oct 30 01:39:20.604363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604175 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12d0d9-345e-4d7d-a874-3adad424a75e" containerName="revoke-keys" Oct 30 01:39:20.604363 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.604185 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0db35f5-5f3a-4d01-b871-c14e81b61471" containerName="istio-proxy" Oct 30 01:39:20.607378 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.607362 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.638445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.638417 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:39:20.638615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.638494 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:39:20.638615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.638556 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:39:20.638615 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.638558 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:39:20.640891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.640869 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p"] Oct 30 01:39:20.649798 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.649771 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bkr\" (UniqueName: \"kubernetes.io/projected/678ff6d7-2f78-4421-9aff-6122c02faa3f-kube-api-access-s7bkr\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.650141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.650106 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.650256 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.650193 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.751301 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.751273 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.751445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.751315 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.751445 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.751366 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bkr\" (UniqueName: \"kubernetes.io/projected/678ff6d7-2f78-4421-9aff-6122c02faa3f-kube-api-access-s7bkr\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.752037 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.752017 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.753837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.753814 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.771949 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.771931 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bkr\" (UniqueName: \"kubernetes.io/projected/678ff6d7-2f78-4421-9aff-6122c02faa3f-kube-api-access-s7bkr\") pod \"maas-api-revoke-sub-452b6d7b-8n29p\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:20.920382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:20.920323 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:21.216418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:21.216396 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p"] Oct 30 01:39:21.216951 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:39:21.216927 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678ff6d7_2f78_4421_9aff_6122c02faa3f.slice/crio-c1a11aad0d1bebf0a2ac7ac8c7ec215f2289f780b18778cfbf9edc6dc45a929c WatchSource:0}: Error finding container c1a11aad0d1bebf0a2ac7ac8c7ec215f2289f780b18778cfbf9edc6dc45a929c: Status 404 returned error can't find the container with id c1a11aad0d1bebf0a2ac7ac8c7ec215f2289f780b18778cfbf9edc6dc45a929c Oct 30 01:39:22.009861 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:22.009823 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" event={"ID":"678ff6d7-2f78-4421-9aff-6122c02faa3f","Type":"ContainerStarted","Data":"e6a2ea3187d22ccc4681f2ac16eb33187814c2f9a2530f17c2fce431dc94310b"} Oct 30 01:39:22.010257 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:22.009888 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" event={"ID":"678ff6d7-2f78-4421-9aff-6122c02faa3f","Type":"ContainerStarted","Data":"c1a11aad0d1bebf0a2ac7ac8c7ec215f2289f780b18778cfbf9edc6dc45a929c"} Oct 30 01:39:22.051668 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:22.051621 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" podStartSLOduration=2.051606231 podStartE2EDuration="2.051606231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:22.050859126 +0000 UTC m=+3646.994032321" watchObservedRunningTime="2026-10-30 01:39:22.051606231 +0000 UTC m=+3646.994779426" Oct 30 01:39:23.020483 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:23.020441 4905 generic.go:356] "Generic (PLEG): container finished" podID="678ff6d7-2f78-4421-9aff-6122c02faa3f" containerID="e6a2ea3187d22ccc4681f2ac16eb33187814c2f9a2530f17c2fce431dc94310b" exitCode=0 Oct 30 01:39:23.021042 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:23.020598 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" event={"ID":"678ff6d7-2f78-4421-9aff-6122c02faa3f","Type":"ContainerDied","Data":"e6a2ea3187d22ccc4681f2ac16eb33187814c2f9a2530f17c2fce431dc94310b"} Oct 30 01:39:24.917700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:24.917663 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:25.108598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.107835 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-xks-ca\") pod \"678ff6d7-2f78-4421-9aff-6122c02faa3f\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " Oct 30 01:39:25.108598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.107926 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-openshift-ca\") pod \"678ff6d7-2f78-4421-9aff-6122c02faa3f\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " Oct 30 01:39:25.108598 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.107966 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678ff6d7-2f78-4421-9aff-6122c02faa3f-kube-api-access-s7bkr\" (UniqueName: \"kubernetes.io/projected/678ff6d7-2f78-4421-9aff-6122c02faa3f-kube-api-access-s7bkr\") pod \"678ff6d7-2f78-4421-9aff-6122c02faa3f\" (UID: \"678ff6d7-2f78-4421-9aff-6122c02faa3f\") " Oct 30 01:39:25.200503 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.199879 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-openshift-ca" pod "678ff6d7-2f78-4421-9aff-6122c02faa3f" (UID: "678ff6d7-2f78-4421-9aff-6122c02faa3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:25.285544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.261205 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:39:25.285544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.261351 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678ff6d7-2f78-4421-9aff-6122c02faa3f-kube-api-access-s7bkr" pod "678ff6d7-2f78-4421-9aff-6122c02faa3f" (UID: "678ff6d7-2f78-4421-9aff-6122c02faa3f"). InnerVolumeSpecName "kube-api-access-s7bkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:25.331544 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.320103 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-xks-ca" pod "678ff6d7-2f78-4421-9aff-6122c02faa3f" (UID: "678ff6d7-2f78-4421-9aff-6122c02faa3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:39:25.374178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.374103 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" Oct 30 01:39:25.377803 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.377782 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678ff6d7-2f78-4421-9aff-6122c02faa3f-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:39:25.385607 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.385571 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bkr\" (UniqueName: \"kubernetes.io/projected/678ff6d7-2f78-4421-9aff-6122c02faa3f-kube-api-access-s7bkr\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:39:25.400281 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.400248 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8n29p" event={"ID":"678ff6d7-2f78-4421-9aff-6122c02faa3f","Type":"ContainerDied","Data":"c1a11aad0d1bebf0a2ac7ac8c7ec215f2289f780b18778cfbf9edc6dc45a929c"} Oct 30 01:39:25.412679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:25.412657 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1a11aad0d1bebf0a2ac7ac8c7ec215f2289f780b18778cfbf9edc6dc45a929c" Oct 30 01:39:25.583553 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:39:25.565326 4905 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678ff6d7_2f78_4421_9aff_6122c02faa3f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:39:38.221825 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.221791 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77"] Oct 30 01:39:38.222434 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.222413 4905 cpu_manager.go:414] "containerMap: removing container" podUID="678ff6d7-2f78-4421-9aff-6122c02faa3f" containerName="revoke-keys" Oct 30 01:39:38.222569 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.222436 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="678ff6d7-2f78-4421-9aff-6122c02faa3f" containerName="revoke-keys" Oct 30 01:39:38.222644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.222611 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="678ff6d7-2f78-4421-9aff-6122c02faa3f" containerName="revoke-keys" Oct 30 01:39:38.225748 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.225726 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.244357 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.244327 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:39:38.257898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.257868 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:39:38.258039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.257892 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:39:38.258039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.257976 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:39:38.261717 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.261694 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77"] Oct 30 01:39:38.305333 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.305230 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.305576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.305348 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.305576 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.305435 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhcc\" (UniqueName: \"kubernetes.io/projected/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-kube-api-access-7zhcc\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.409119 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.409072 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.409325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.409154 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.409325 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.409227 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhcc\" (UniqueName: \"kubernetes.io/projected/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-kube-api-access-7zhcc\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.410545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.410329 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.421565 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.421222 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.440039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.433376 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhcc\" (UniqueName: \"kubernetes.io/projected/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-kube-api-access-7zhcc\") pod \"maas-api-revoke-sub-3e31a506-47v77\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.539655 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.539617 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:38.884141 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:39:38.884091 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab8c02f_30d7_45f8_ac96_e4636f33f37c.slice/crio-54eb22bc1ed89ed934f88d389434b83616793fc8a77071a22b4166b265fe86a3 WatchSource:0}: Error finding container 54eb22bc1ed89ed934f88d389434b83616793fc8a77071a22b4166b265fe86a3: Status 404 returned error can't find the container with id 54eb22bc1ed89ed934f88d389434b83616793fc8a77071a22b4166b265fe86a3 Oct 30 01:39:38.892880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:38.892856 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77"] Oct 30 01:39:39.518214 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:39.518182 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" event={"ID":"7ab8c02f-30d7-45f8-ac96-e4636f33f37c","Type":"ContainerStarted","Data":"d654da402fedf8c64d5dbb6cbbfd1b777825a3e34400acc25198c2f5eeb508e4"} Oct 30 01:39:39.518794 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:39.518762 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" event={"ID":"7ab8c02f-30d7-45f8-ac96-e4636f33f37c","Type":"ContainerStarted","Data":"54eb22bc1ed89ed934f88d389434b83616793fc8a77071a22b4166b265fe86a3"} Oct 30 01:39:39.601554 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:39.595921 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" podStartSLOduration=1.595901872 podStartE2EDuration="1.595901872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:39.576708041 +0000 UTC m=+3664.519881235" watchObservedRunningTime="2026-10-30 01:39:39.595901872 +0000 UTC m=+3664.539075066" Oct 30 01:39:40.555718 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:40.555666 4905 generic.go:356] "Generic (PLEG): container finished" podID="7ab8c02f-30d7-45f8-ac96-e4636f33f37c" containerID="d654da402fedf8c64d5dbb6cbbfd1b777825a3e34400acc25198c2f5eeb508e4" exitCode=0 Oct 30 01:39:40.556362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:40.556337 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" event={"ID":"7ab8c02f-30d7-45f8-ac96-e4636f33f37c","Type":"ContainerDied","Data":"d654da402fedf8c64d5dbb6cbbfd1b777825a3e34400acc25198c2f5eeb508e4"} Oct 30 01:39:41.700300 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.700274 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:41.771132 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.771097 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-xks-ca\") pod \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " Oct 30 01:39:41.771274 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.771214 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-kube-api-access-7zhcc\" (UniqueName: \"kubernetes.io/projected/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-kube-api-access-7zhcc\") pod \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " Oct 30 01:39:41.771348 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.771331 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-openshift-ca\") pod \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\" (UID: \"7ab8c02f-30d7-45f8-ac96-e4636f33f37c\") " Oct 30 01:39:41.771771 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.771739 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-openshift-ca" pod "7ab8c02f-30d7-45f8-ac96-e4636f33f37c" (UID: "7ab8c02f-30d7-45f8-ac96-e4636f33f37c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:41.771896 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.771839 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:39:41.773451 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.773422 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-kube-api-access-7zhcc" pod "7ab8c02f-30d7-45f8-ac96-e4636f33f37c" (UID: "7ab8c02f-30d7-45f8-ac96-e4636f33f37c"). InnerVolumeSpecName "kube-api-access-7zhcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:41.773602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.773580 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-xks-ca" pod "7ab8c02f-30d7-45f8-ac96-e4636f33f37c" (UID: "7ab8c02f-30d7-45f8-ac96-e4636f33f37c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:39:41.872654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.872594 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:39:41.872654 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:41.872621 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhcc\" (UniqueName: \"kubernetes.io/projected/7ab8c02f-30d7-45f8-ac96-e4636f33f37c-kube-api-access-7zhcc\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:39:42.419260 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:39:42.419225 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:39:42.569602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:42.569563 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" event={"ID":"7ab8c02f-30d7-45f8-ac96-e4636f33f37c","Type":"ContainerDied","Data":"54eb22bc1ed89ed934f88d389434b83616793fc8a77071a22b4166b265fe86a3"} Oct 30 01:39:42.569602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:42.569593 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-47v77" Oct 30 01:39:42.569822 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:42.569626 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54eb22bc1ed89ed934f88d389434b83616793fc8a77071a22b4166b265fe86a3" Oct 30 01:39:43.029605 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:43.029571 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc"] Oct 30 01:39:43.044475 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:43.044440 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58e54186-prxhc"] Oct 30 01:39:43.189687 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:43.189652 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb5a0d6-b402-4d35-92e8-2f63681b34d8" path="/var/lib/kubelet/pods/0eb5a0d6-b402-4d35-92e8-2f63681b34d8/volumes" Oct 30 01:39:47.692338 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:47.692304 4905 scope.go:122] "RemoveContainer" containerID="bece83a414c7aed065fe42ee6bc85efb9e236fbfff2b316bf1ae5d6809abf6a8" Oct 30 01:39:47.701892 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:47.701871 4905 scope.go:122] "RemoveContainer" containerID="b4d5f4f1b6eed1c8cb6d995760a8151d624777a8a0f5e2071941d719ea7fd6c8" Oct 30 01:39:47.710587 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:39:47.710567 4905 scope.go:122] "RemoveContainer" containerID="8fd85e5ab5d6e4c0d821fe44a392dce8293702c90216d76cbab80be14b5ebba2" Oct 30 01:40:01.797493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.797458 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6"] Oct 30 01:40:01.798024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.797907 4905 cpu_manager.go:414] "containerMap: removing container" podUID="7ab8c02f-30d7-45f8-ac96-e4636f33f37c" containerName="revoke-keys" Oct 30 01:40:01.798024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.797918 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab8c02f-30d7-45f8-ac96-e4636f33f37c" containerName="revoke-keys" Oct 30 01:40:01.798024 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.797988 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab8c02f-30d7-45f8-ac96-e4636f33f37c" containerName="revoke-keys" Oct 30 01:40:01.802938 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.802923 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:01.822657 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.822632 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6"] Oct 30 01:40:01.833212 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.833187 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:40:01.833337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.833245 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:40:01.833337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.833273 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:40:01.833337 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.833328 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\"" Oct 30 01:40:01.931198 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.931171 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:01.931358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.931322 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:01.931358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:01.931347 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srb74\" (UniqueName: \"kubernetes.io/projected/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-kube-api-access-srb74\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.032148 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.032108 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.032324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.032271 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.032324 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.032299 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srb74\" (UniqueName: \"kubernetes.io/projected/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-kube-api-access-srb74\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.032945 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.032925 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.034837 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.034813 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.053487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.053414 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srb74\" (UniqueName: \"kubernetes.io/projected/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-kube-api-access-srb74\") pod \"maas-api-revoke-sub-04847998-pzks6\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.116320 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.116289 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:02.410217 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.410185 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6"] Oct 30 01:40:02.410383 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:40:02.410361 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae4fdc6_d51b_4fc3_8954_99bc4c907e88.slice/crio-8a75fa2776cebb5e7ae7b85587b74748b37401e53799487db751d8b1d71a2197 WatchSource:0}: Error finding container 8a75fa2776cebb5e7ae7b85587b74748b37401e53799487db751d8b1d71a2197: Status 404 returned error can't find the container with id 8a75fa2776cebb5e7ae7b85587b74748b37401e53799487db751d8b1d71a2197 Oct 30 01:40:02.739315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:02.739285 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" event={"ID":"eae4fdc6-d51b-4fc3-8954-99bc4c907e88","Type":"ContainerStarted","Data":"8a75fa2776cebb5e7ae7b85587b74748b37401e53799487db751d8b1d71a2197"} Oct 30 01:40:03.746089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:03.746053 4905 generic.go:356] "Generic (PLEG): container finished" podID="eae4fdc6-d51b-4fc3-8954-99bc4c907e88" containerID="0e4e8c4e9087caf23bde1e717dfe0667560ed67d858c724a546cce60dc60a0d5" exitCode=0 Oct 30 01:40:03.746563 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:03.746132 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" event={"ID":"eae4fdc6-d51b-4fc3-8954-99bc4c907e88","Type":"ContainerDied","Data":"0e4e8c4e9087caf23bde1e717dfe0667560ed67d858c724a546cce60dc60a0d5"} Oct 30 01:40:04.893272 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:04.893249 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:04.959236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:04.959201 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-xks-ca\") pod \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " Oct 30 01:40:04.959410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:04.959246 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-openshift-ca\") pod \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " Oct 30 01:40:04.959410 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:04.959301 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-kube-api-access-srb74\" (UniqueName: \"kubernetes.io/projected/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-kube-api-access-srb74\") pod \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\" (UID: \"eae4fdc6-d51b-4fc3-8954-99bc4c907e88\") " Oct 30 01:40:04.959728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:04.959696 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-openshift-ca" pod "eae4fdc6-d51b-4fc3-8954-99bc4c907e88" (UID: "eae4fdc6-d51b-4fc3-8954-99bc4c907e88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:04.961550 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:04.961504 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-xks-ca" pod "eae4fdc6-d51b-4fc3-8954-99bc4c907e88" (UID: "eae4fdc6-d51b-4fc3-8954-99bc4c907e88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:40:04.961714 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:04.961688 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-kube-api-access-srb74" pod "eae4fdc6-d51b-4fc3-8954-99bc4c907e88" (UID: "eae4fdc6-d51b-4fc3-8954-99bc4c907e88"). InnerVolumeSpecName "kube-api-access-srb74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:05.060891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:05.060813 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-xks-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:40:05.060891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:05.060841 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-maas-api-openshift-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:40:05.060891 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:05.060852 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srb74\" (UniqueName: \"kubernetes.io/projected/eae4fdc6-d51b-4fc3-8954-99bc4c907e88-kube-api-access-srb74\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:40:05.758372 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:05.758335 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" Oct 30 01:40:05.758571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:05.758336 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04847998-pzks6" event={"ID":"eae4fdc6-d51b-4fc3-8954-99bc4c907e88","Type":"ContainerDied","Data":"8a75fa2776cebb5e7ae7b85587b74748b37401e53799487db751d8b1d71a2197"} Oct 30 01:40:05.758571 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:05.758461 4905 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a75fa2776cebb5e7ae7b85587b74748b37401e53799487db751d8b1d71a2197" Oct 30 01:40:12.415354 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:40:12.415316 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:40:13.899747 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.899711 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ssn64"] Oct 30 01:40:13.900295 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.900277 4905 cpu_manager.go:414] "containerMap: removing container" podUID="eae4fdc6-d51b-4fc3-8954-99bc4c907e88" containerName="revoke-keys" Oct 30 01:40:13.900346 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.900297 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae4fdc6-d51b-4fc3-8954-99bc4c907e88" containerName="revoke-keys" Oct 30 01:40:13.900392 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.900382 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae4fdc6-d51b-4fc3-8954-99bc4c907e88" containerName="revoke-keys" Oct 30 01:40:13.903828 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.903803 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:13.938359 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.938319 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:40:13.941097 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.941070 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ssn64"] Oct 30 01:40:13.941245 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.941185 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:40:13.941411 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.941385 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 01:40:13.941482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.941452 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:40:13.952361 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:13.952336 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6w4xf\"" Oct 30 01:40:14.026562 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.026500 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h"] Oct 30 01:40:14.028460 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.028440 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s27h"] Oct 30 01:40:14.047382 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.047352 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-maas-api-tls\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.047512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.047412 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-metrics-certs\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.047512 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.047495 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw44\" (UniqueName: \"kubernetes.io/projected/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-kube-api-access-cfw44\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.047631 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.047614 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-openshift-service-ca\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.149089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.149056 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-metrics-certs\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.149263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.149096 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw44\" (UniqueName: \"kubernetes.io/projected/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-kube-api-access-cfw44\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.149263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.149144 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-openshift-service-ca\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.149263 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.149199 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-maas-api-tls\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.149884 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.149818 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-openshift-service-ca\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.151692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.151675 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-metrics-certs\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.151765 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.151705 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-maas-api-tls\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.175842 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.175817 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw44\" (UniqueName: \"kubernetes.io/projected/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-kube-api-access-cfw44\") pod \"maas-api-5c48877c75-ssn64\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.217970 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.217942 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:14.516906 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.516877 4905 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:40:14.519159 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.519070 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ssn64"] Oct 30 01:40:14.806179 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:14.806097 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" event={"ID":"59a936d9-2a68-4e8a-a4c3-bd87c416cb59","Type":"ContainerStarted","Data":"0e3d87e95e27bafdbfe4efdcbdf123ca532ad6474dab6cb30de2b7d829566787"} Oct 30 01:40:15.190077 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:15.190004 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592" path="/var/lib/kubelet/pods/4e00a2a3-e8c3-4ea8-9a7e-5828ca03f592/volumes" Oct 30 01:40:15.812854 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:15.812812 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" event={"ID":"59a936d9-2a68-4e8a-a4c3-bd87c416cb59","Type":"ContainerStarted","Data":"bc0d22f25d9d87c1aa8b8100aabe504f2c33e7986ca02c31516fd0f54ec95273"} Oct 30 01:40:15.813039 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:15.812871 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:15.859389 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:15.859339 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" podStartSLOduration=2.505197592 podStartE2EDuration="2.85932546s" totalImagesPullingTime="354.127868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:15.858093068 +0000 UTC m=+3700.801266261" watchObservedRunningTime="2026-10-30 01:40:15.85932546 +0000 UTC m=+3700.802498653" Oct 30 01:40:21.823852 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:21.823826 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:22.028072 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.028034 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2s97h"] Oct 30 01:40:22.032878 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.032857 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.050603 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.050577 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2s97h"] Oct 30 01:40:22.126751 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.126655 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvm74\" (UniqueName: \"kubernetes.io/projected/512d5c9e-eeca-4a61-9b24-4128565698cb-kube-api-access-dvm74\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.126939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.126798 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512d5c9e-eeca-4a61-9b24-4128565698cb-maas-api-tls\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.126939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.126826 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/512d5c9e-eeca-4a61-9b24-4128565698cb-metrics-certs\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.126939 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.126886 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d5c9e-eeca-4a61-9b24-4128565698cb-openshift-service-ca\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.227956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.227925 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512d5c9e-eeca-4a61-9b24-4128565698cb-maas-api-tls\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.227956 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.227956 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/512d5c9e-eeca-4a61-9b24-4128565698cb-metrics-certs\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.228236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.227984 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d5c9e-eeca-4a61-9b24-4128565698cb-openshift-service-ca\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.228236 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.228032 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvm74\" (UniqueName: \"kubernetes.io/projected/512d5c9e-eeca-4a61-9b24-4128565698cb-kube-api-access-dvm74\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.228658 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.228631 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d5c9e-eeca-4a61-9b24-4128565698cb-openshift-service-ca\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.230812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.230785 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/512d5c9e-eeca-4a61-9b24-4128565698cb-metrics-certs\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.230913 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.230818 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512d5c9e-eeca-4a61-9b24-4128565698cb-maas-api-tls\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.248702 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.248674 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvm74\" (UniqueName: \"kubernetes.io/projected/512d5c9e-eeca-4a61-9b24-4128565698cb-kube-api-access-dvm74\") pod \"maas-api-595cc648c-2s97h\" (UID: \"512d5c9e-eeca-4a61-9b24-4128565698cb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.348113 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.348085 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:22.647418 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.647387 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2s97h"] Oct 30 01:40:22.647673 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:40:22.647653 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512d5c9e_eeca_4a61_9b24_4128565698cb.slice/crio-9eaa1bcd8d68b6a89e65f969ccddd294609f0132ba1261241671063d986c641e WatchSource:0}: Error finding container 9eaa1bcd8d68b6a89e65f969ccddd294609f0132ba1261241671063d986c641e: Status 404 returned error can't find the container with id 9eaa1bcd8d68b6a89e65f969ccddd294609f0132ba1261241671063d986c641e Oct 30 01:40:22.853597 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:22.853557 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" event={"ID":"512d5c9e-eeca-4a61-9b24-4128565698cb","Type":"ContainerStarted","Data":"9eaa1bcd8d68b6a89e65f969ccddd294609f0132ba1261241671063d986c641e"} Oct 30 01:40:23.915545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:23.915463 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" event={"ID":"512d5c9e-eeca-4a61-9b24-4128565698cb","Type":"ContainerStarted","Data":"bbebc2232b6b7d6b07e70097385321654d7872399b3f2b8e3d1f8727d90fcff6"} Oct 30 01:40:23.920541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:23.916670 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:23.977541 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:23.976973 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" podStartSLOduration=2.633167926 podStartE2EDuration="2.976954873s" totalImagesPullingTime="343.786947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:23.974882471 +0000 UTC m=+3708.918055667" watchObservedRunningTime="2026-10-30 01:40:23.976954873 +0000 UTC m=+3708.920128069" Oct 30 01:40:30.944158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:30.944127 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-2s97h" Oct 30 01:40:31.016877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:31.016826 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ssn64"] Oct 30 01:40:31.017440 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:31.017391 4905 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" podUID="59a936d9-2a68-4e8a-a4c3-bd87c416cb59" containerName="maas-api" containerID="cri-o://bc0d22f25d9d87c1aa8b8100aabe504f2c33e7986ca02c31516fd0f54ec95273" gracePeriod=30 Oct 30 01:40:31.819076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:31.819030 4905 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" podUID="59a936d9-2a68-4e8a-a4c3-bd87c416cb59" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.107:8443/health\": dial tcp 10.132.1.107:8443: connect: connection refused" Oct 30 01:40:33.048545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.047393 4905 generic.go:356] "Generic (PLEG): container finished" podID="59a936d9-2a68-4e8a-a4c3-bd87c416cb59" containerID="bc0d22f25d9d87c1aa8b8100aabe504f2c33e7986ca02c31516fd0f54ec95273" exitCode=0 Oct 30 01:40:33.048545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.047507 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" event={"ID":"59a936d9-2a68-4e8a-a4c3-bd87c416cb59","Type":"ContainerDied","Data":"bc0d22f25d9d87c1aa8b8100aabe504f2c33e7986ca02c31516fd0f54ec95273"} Oct 30 01:40:33.129323 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.129295 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:33.299602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.298811 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-kube-api-access-cfw44\" (UniqueName: \"kubernetes.io/projected/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-kube-api-access-cfw44\") pod \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " Oct 30 01:40:33.299602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.298869 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-openshift-service-ca\") pod \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " Oct 30 01:40:33.299602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.299037 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-maas-api-tls\") pod \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " Oct 30 01:40:33.299602 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.299098 4905 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-metrics-certs\") pod \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\" (UID: \"59a936d9-2a68-4e8a-a4c3-bd87c416cb59\") " Oct 30 01:40:33.324566 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.323704 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-openshift-service-ca" pod "59a936d9-2a68-4e8a-a4c3-bd87c416cb59" (UID: "59a936d9-2a68-4e8a-a4c3-bd87c416cb59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:33.397899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.397698 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-metrics-certs" pod "59a936d9-2a68-4e8a-a4c3-bd87c416cb59" (UID: "59a936d9-2a68-4e8a-a4c3-bd87c416cb59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:40:33.397899 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.397730 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-kube-api-access-cfw44" pod "59a936d9-2a68-4e8a-a4c3-bd87c416cb59" (UID: "59a936d9-2a68-4e8a-a4c3-bd87c416cb59"). InnerVolumeSpecName "kube-api-access-cfw44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:33.398149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.397943 4905 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-maas-api-tls" pod "59a936d9-2a68-4e8a-a4c3-bd87c416cb59" (UID: "59a936d9-2a68-4e8a-a4c3-bd87c416cb59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:40:33.426880 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.426850 4905 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-maas-api-tls\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:40:33.427135 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.427117 4905 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-metrics-certs\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:40:33.427286 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.427267 4905 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw44\" (UniqueName: \"kubernetes.io/projected/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-kube-api-access-cfw44\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:40:33.427416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:33.427401 4905 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a936d9-2a68-4e8a-a4c3-bd87c416cb59-openshift-service-ca\") on node \"ip-10-0-130-171.ec2.internal\" DevicePath \"\"" Oct 30 01:40:34.110545 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:34.110467 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" event={"ID":"59a936d9-2a68-4e8a-a4c3-bd87c416cb59","Type":"ContainerDied","Data":"0e3d87e95e27bafdbfe4efdcbdf123ca532ad6474dab6cb30de2b7d829566787"} Oct 30 01:40:34.111614 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:34.111563 4905 scope.go:122] "RemoveContainer" containerID="bc0d22f25d9d87c1aa8b8100aabe504f2c33e7986ca02c31516fd0f54ec95273" Oct 30 01:40:34.112194 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:34.112175 4905 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ssn64" Oct 30 01:40:34.175666 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:34.175636 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ssn64"] Oct 30 01:40:34.189244 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:34.189215 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ssn64"] Oct 30 01:40:35.190252 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:35.190215 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a936d9-2a68-4e8a-a4c3-bd87c416cb59" path="/var/lib/kubelet/pods/59a936d9-2a68-4e8a-a4c3-bd87c416cb59/volumes" Oct 30 01:40:42.414350 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:40:42.414315 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:40:47.761596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:40:47.760684 4905 scope.go:122] "RemoveContainer" containerID="7f76225f62323bd9e80b023f1844f84311f656385b321a2c47dab32efd634625" Oct 30 01:41:12.557735 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:41:12.557691 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:41:22.156149 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:22.156117 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5d8c498c66-26f4b_53ff45f8-5864-4727-b72c-b63399c3657c/manager/0.log" Oct 30 01:41:23.040177 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:23.040143 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d6f77c9d-7kkl4_972beb40-dba6-4ac0-9848-94188dadbe68/manager/0.log" Oct 30 01:41:24.299051 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:24.299021 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-wzp4q_ee7f9e1a-0987-4812-bfba-97cad9512ab1/copy-manifests/0.log" Oct 30 01:41:24.341916 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:24.341882 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-wzp4q_ee7f9e1a-0987-4812-bfba-97cad9512ab1/manager/0.log" Oct 30 01:41:24.543617 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:24.543588 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-25kxk_cccc3e6c-ecee-4d0f-8719-1075e1789e73/proxy/0.log" Oct 30 01:41:25.287778 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:25.287742 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-2s97h_512d5c9e-eeca-4a61-9b24-4128565698cb/maas-api/0.log" Oct 30 01:41:25.811504 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:25.811469 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-04847998-pzks6_eae4fdc6-d51b-4fc3-8954-99bc4c907e88/revoke-keys/0.log" Oct 30 01:41:25.989211 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:25.989182 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-36288509-9h5st_fbd6957f-30b3-473d-b6cc-38eef899c898/revoke-keys/0.log" Oct 30 01:41:26.169416 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:26.169353 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-47v77_7ab8c02f-30d7-45f8-ac96-e4636f33f37c/revoke-keys/0.log" Oct 30 01:41:26.336029 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:26.336009 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-8n29p_678ff6d7-2f78-4421-9aff-6122c02faa3f/revoke-keys/0.log" Oct 30 01:41:26.724578 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:26.724551 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-fbhc9_ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6/revoke-keys/0.log" Oct 30 01:41:26.896395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:26.896365 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-djqqq_d8dc2195-abf2-4712-8913-1201fbdd9461/revoke-keys/0.log" Oct 30 01:41:29.501089 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:29.501060 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-5lhrk_25adc5e3-09cf-4499-beef-1c6fe31e1706/manager/0.log" Oct 30 01:41:29.909190 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:29.909111 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-nh5jh_7b06bf07-f94a-4f87-af61-69f35a22aeb2/manager/0.log" Oct 30 01:41:30.656429 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:30.656398 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d54frhj_9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a/istio-proxy/0.log" Oct 30 01:41:30.848136 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:30.848105 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8rf4k_26e2681e-21d4-42a6-967c-d462361c99e6/discovery/0.log" Oct 30 01:41:31.036844 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:31.036817 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7597b65cbf-jb59g_f9305cf4-d3b1-43cc-b248-33a8d2bbca5c/kube-auth-proxy/0.log" Oct 30 01:41:31.414482 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:31.414394 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-2rs7m_2e721927-370e-4f47-9eef-60330ef5886a/istio-proxy/0.log" Oct 30 01:41:31.590634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:31.590604 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-2z245_b1988474-6586-4686-b98d-34a146461f5e/istio-proxy/0.log" Oct 30 01:41:32.165708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:32.165675 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hzfrs_f9b68d31-1fc7-4045-9167-11e44e372333/istio-proxy/0.log" Oct 30 01:41:32.745959 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:32.745923 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-pwsj2_e5fe03d5-2065-4957-a660-22bd064b5892/istio-proxy/0.log" Oct 30 01:41:33.326947 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:33.326913 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-t8bwv_3cd3b9f8-446f-4086-be0a-1754d63dc090/payload-pre-processing/0.log" Oct 30 01:41:33.506850 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:33.506804 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:41:33.693584 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:33.693486 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7844cf9dc7-sv5gn_fc3bbd76-9720-4475-8a2f-c8bee23c004b/router/0.log" Oct 30 01:41:34.423303 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:34.423275 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g_38c45736-0323-495f-8968-2a96fe77d026/main/0.log" Oct 30 01:41:34.604762 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:34.604735 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2rzg2_20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec/main/0.log" Oct 30 01:41:35.147708 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:35.147661 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm_34ba07ff-2d49-4483-a644-04ce768e4937/main/0.log" Oct 30 01:41:35.327141 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:35.327115 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9_8e372dca-3376-4d81-9285-f692259df304/main/0.log" Oct 30 01:41:42.414512 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:41:42.414480 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:41:45.135090 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:45.135055 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5d8c498c66-26f4b_53ff45f8-5864-4727-b72c-b63399c3657c/manager/0.log" Oct 30 01:41:45.959332 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:45.959297 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d6f77c9d-7kkl4_972beb40-dba6-4ac0-9848-94188dadbe68/manager/0.log" Oct 30 01:41:47.224871 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:47.224838 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-wzp4q_ee7f9e1a-0987-4812-bfba-97cad9512ab1/copy-manifests/0.log" Oct 30 01:41:47.266924 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:47.266889 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-wzp4q_ee7f9e1a-0987-4812-bfba-97cad9512ab1/manager/0.log" Oct 30 01:41:47.484978 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:47.484946 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-25kxk_cccc3e6c-ecee-4d0f-8719-1075e1789e73/proxy/0.log" Oct 30 01:41:47.900175 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:47.900109 4905 scope.go:122] "RemoveContainer" containerID="127b7b3a8a3185266821945303ae4cb97715dcb004692f3643c6544d43214162" Oct 30 01:41:47.909596 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:47.909571 4905 scope.go:122] "RemoveContainer" containerID="07e8a62c48d1064eb59f7f09b465f0a9319f950f3d82093ebcb329812a03a62e" Oct 30 01:41:48.005672 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:48.005650 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-2s97h_512d5c9e-eeca-4a61-9b24-4128565698cb/maas-api/0.log" Oct 30 01:41:48.536248 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:48.536222 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-04847998-pzks6_eae4fdc6-d51b-4fc3-8954-99bc4c907e88/revoke-keys/0.log" Oct 30 01:41:48.708682 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:48.708650 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-36288509-9h5st_fbd6957f-30b3-473d-b6cc-38eef899c898/revoke-keys/0.log" Oct 30 01:41:48.893312 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:48.893251 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-47v77_7ab8c02f-30d7-45f8-ac96-e4636f33f37c/revoke-keys/0.log" Oct 30 01:41:49.068799 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:49.068780 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-8n29p_678ff6d7-2f78-4421-9aff-6122c02faa3f/revoke-keys/0.log" Oct 30 01:41:49.416170 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:49.416151 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-fbhc9_ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6/revoke-keys/0.log" Oct 30 01:41:49.589078 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:49.589056 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-djqqq_d8dc2195-abf2-4712-8913-1201fbdd9461/revoke-keys/0.log" Oct 30 01:41:51.833580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:51.833547 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-5lhrk_25adc5e3-09cf-4499-beef-1c6fe31e1706/manager/0.log" Oct 30 01:41:52.236292 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:52.236268 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-nh5jh_7b06bf07-f94a-4f87-af61-69f35a22aeb2/manager/0.log" Oct 30 01:41:52.752493 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:52.752461 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d54frhj_9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a/istio-proxy/0.log" Oct 30 01:41:52.939700 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:52.939677 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8rf4k_26e2681e-21d4-42a6-967c-d462361c99e6/discovery/0.log" Oct 30 01:41:53.110644 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:53.110587 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7597b65cbf-jb59g_f9305cf4-d3b1-43cc-b248-33a8d2bbca5c/kube-auth-proxy/0.log" Oct 30 01:41:53.465435 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:53.465368 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-2rs7m_2e721927-370e-4f47-9eef-60330ef5886a/istio-proxy/0.log" Oct 30 01:41:53.648772 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:53.648743 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-2z245_b1988474-6586-4686-b98d-34a146461f5e/istio-proxy/0.log" Oct 30 01:41:54.200781 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:54.200755 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hzfrs_f9b68d31-1fc7-4045-9167-11e44e372333/istio-proxy/0.log" Oct 30 01:41:54.790350 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:54.790319 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-pwsj2_e5fe03d5-2065-4957-a660-22bd064b5892/istio-proxy/0.log" Oct 30 01:41:55.390454 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:55.390427 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-t8bwv_3cd3b9f8-446f-4086-be0a-1754d63dc090/payload-pre-processing/0.log" Oct 30 01:41:55.568877 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:55.568851 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:41:55.758458 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:55.758430 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7844cf9dc7-sv5gn_fc3bbd76-9720-4475-8a2f-c8bee23c004b/router/0.log" Oct 30 01:41:56.285634 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:56.285609 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5t69g_38c45736-0323-495f-8968-2a96fe77d026/main/0.log" Oct 30 01:41:56.466558 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:56.466507 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2rzg2_20db7a35-d107-4bda-a4ce-5f7d7a5ae6ec/main/0.log" Oct 30 01:41:57.015181 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:57.015117 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sc4bm_34ba07ff-2d49-4483-a644-04ce768e4937/main/0.log" Oct 30 01:41:57.192358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:41:57.192334 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7d9v9_8e372dca-3376-4d81-9285-f692259df304/main/0.log" Oct 30 01:42:03.029326 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:03.029286 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq"] Oct 30 01:42:03.044081 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:03.044052 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-djqqq"] Oct 30 01:42:03.188898 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:03.188862 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8dc2195-abf2-4712-8913-1201fbdd9461" path="/var/lib/kubelet/pods/d8dc2195-abf2-4712-8913-1201fbdd9461/volumes" Oct 30 01:42:12.413997 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:42:12.413965 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:42:19.026926 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:19.026844 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9"] Oct 30 01:42:19.040235 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:19.040202 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fbhc9"] Oct 30 01:42:19.189174 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:19.189146 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6" path="/var/lib/kubelet/pods/ce01099e-9bd9-4c60-ab15-7bf1cd6a59d6/volumes" Oct 30 01:42:27.345763 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:27.345734 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d6f77c9d-7kkl4_972beb40-dba6-4ac0-9848-94188dadbe68/manager/0.log" Oct 30 01:42:27.666716 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:27.666636 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d6f77c9d-7kkl4_972beb40-dba6-4ac0-9848-94188dadbe68/manager/0.log" Oct 30 01:42:28.303158 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:28.303130 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:42:28.481606 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:42:28.481581 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:42:42.415009 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:42:42.414973 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:43:06.275955 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:06.275923 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d6f77c9d-7kkl4_972beb40-dba6-4ac0-9848-94188dadbe68/manager/0.log" Oct 30 01:43:06.587087 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:06.586994 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d6f77c9d-7kkl4_972beb40-dba6-4ac0-9848-94188dadbe68/manager/0.log" Oct 30 01:43:07.223688 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:07.223664 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:43:07.405396 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:07.405372 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9tr26_3a48816a-4625-4906-b1a2-9876cb2e1365/payload-processing/0.log" Oct 30 01:43:12.025553 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:12.025496 4905 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st"] Oct 30 01:43:12.039004 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:12.038975 4905 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36288509-9h5st"] Oct 30 01:43:12.173679 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:12.173653 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vctgv_447c7c04-4097-4065-9e32-ac6b6dc15706/global-pull-secret-syncer/0.log" Oct 30 01:43:12.413596 ip-10-0-130-171 kubenswrapper[4905]: E1030 01:43:12.413505 4905 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkhms" containerName="extractor" Oct 30 01:43:12.501246 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:12.501224 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nzzvk_7b1ecc80-c5e0-40ac-b11c-957168289d3b/connection-checker/0.log" Oct 30 01:43:12.615315 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:12.615293 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wl87c_2968e47e-4347-42e8-8944-6415c8cef439/connection-checker/0.log" Oct 30 01:43:12.728775 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:12.728717 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zbr9f_6334ee19-9ae2-42e9-b704-39213b81e260/connection-checker/0.log" Oct 30 01:43:13.035206 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:13.035173 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8f8v5_ee3561da-40f3-40db-a14b-ad558c5236d7/konnectivity-agent/0.log" Oct 30 01:43:13.189288 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:13.189259 4905 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd6957f-30b3-473d-b6cc-38eef899c898" path="/var/lib/kubelet/pods/fbd6957f-30b3-473d-b6cc-38eef899c898/volumes" Oct 30 01:43:13.290400 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:13.290306 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-171.ec2.internal_92d13e825dfc8d10deb63823b2be0da4/haproxy/0.log" Oct 30 01:43:20.812897 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:20.812853 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-5lhrk_25adc5e3-09cf-4499-beef-1c6fe31e1706/manager/0.log" Oct 30 01:43:21.089202 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:21.089098 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-nh5jh_7b06bf07-f94a-4f87-af61-69f35a22aeb2/manager/0.log" Oct 30 01:43:22.309572 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.309540 4905 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9"] Oct 30 01:43:22.309980 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.309966 4905 cpu_manager.go:414] "containerMap: removing container" podUID="59a936d9-2a68-4e8a-a4c3-bd87c416cb59" containerName="maas-api" Oct 30 01:43:22.310031 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.309981 4905 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a936d9-2a68-4e8a-a4c3-bd87c416cb59" containerName="maas-api" Oct 30 01:43:22.310076 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.310067 4905 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a936d9-2a68-4e8a-a4c3-bd87c416cb59" containerName="maas-api" Oct 30 01:43:22.313419 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.313399 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.332914 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.332889 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9"] Oct 30 01:43:22.400017 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.399989 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-proc\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.400134 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.400117 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-sys\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.400178 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.400163 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-lib-modules\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.400216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.400180 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-podres\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.400216 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.400199 4905 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65q9\" (UniqueName: \"kubernetes.io/projected/7d5e5e42-cf96-4b14-b860-1e43062416cf-kube-api-access-h65q9\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.420903 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.420874 4905 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dw966\"/\"default-dockercfg-rnrqh\"" Oct 30 01:43:22.421009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.420925 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dw966\"/\"kube-root-ca.crt\"" Oct 30 01:43:22.421009 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.420949 4905 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dw966\"/\"openshift-service-ca.crt\"" Oct 30 01:43:22.500575 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500545 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-proc\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500623 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-sys\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500647 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-lib-modules\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500664 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-podres\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500673 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-proc\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500692 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500683 4905 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65q9\" (UniqueName: \"kubernetes.io/projected/7d5e5e42-cf96-4b14-b860-1e43062416cf-kube-api-access-h65q9\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500746 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-podres\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500755 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-sys\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.500918 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.500784 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d5e5e42-cf96-4b14-b860-1e43062416cf-lib-modules\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.520882 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.520860 4905 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65q9\" (UniqueName: \"kubernetes.io/projected/7d5e5e42-cf96-4b14-b860-1e43062416cf-kube-api-access-h65q9\") pod \"perf-node-gather-daemonset-r9bw9\" (UID: \"7d5e5e42-cf96-4b14-b860-1e43062416cf\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.626484 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.626435 4905 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:22.920814 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:22.920784 4905 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9"] Oct 30 01:43:22.921391 ip-10-0-130-171 kubenswrapper[4905]: W1030 01:43:22.921354 4905 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7d5e5e42_cf96_4b14_b860_1e43062416cf.slice/crio-0f63b6d3572219001c14974648bf5f0fbc94904c0007b2bc365e820c54206e5a WatchSource:0}: Error finding container 0f63b6d3572219001c14974648bf5f0fbc94904c0007b2bc365e820c54206e5a: Status 404 returned error can't find the container with id 0f63b6d3572219001c14974648bf5f0fbc94904c0007b2bc365e820c54206e5a Oct 30 01:43:23.090448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:23.090417 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" event={"ID":"7d5e5e42-cf96-4b14-b860-1e43062416cf","Type":"ContainerStarted","Data":"84087945e15ade0ded7c5610f7352617f6590d7425c4c4e681188dde26ef6934"} Oct 30 01:43:23.090580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:23.090474 4905 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" event={"ID":"7d5e5e42-cf96-4b14-b860-1e43062416cf","Type":"ContainerStarted","Data":"0f63b6d3572219001c14974648bf5f0fbc94904c0007b2bc365e820c54206e5a"} Oct 30 01:43:23.090580 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:23.090502 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:23.131358 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:23.131320 4905 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" podStartSLOduration=1.131308113 podStartE2EDuration="1.131308113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:23.130919233 +0000 UTC m=+3888.074092426" watchObservedRunningTime="2026-10-30 01:43:23.131308113 +0000 UTC m=+3888.074481361" Oct 30 01:43:23.182142 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:23.182093 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6xspg_f8053370-48fa-4a20-8865-96d61d33a0b4/cluster-monitoring-operator/0.log" Oct 30 01:43:23.824864 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:23.824832 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29t9j_cf9434e3-f9aa-428a-83b2-3adb3518cf0b/node-exporter/0.log" Oct 30 01:43:23.932701 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:23.932675 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29t9j_cf9434e3-f9aa-428a-83b2-3adb3518cf0b/kube-rbac-proxy/0.log" Oct 30 01:43:24.041368 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:24.041347 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29t9j_cf9434e3-f9aa-428a-83b2-3adb3518cf0b/init-textfile/0.log" Oct 30 01:43:28.333754 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:28.333711 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-vsv6w_abb2b77c-ea98-49f5-a77d-f531c74980c0/networking-console-plugin/0.log" Oct 30 01:43:29.108395 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:29.108367 4905 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-r9bw9" Oct 30 01:43:29.257448 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:29.257422 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:43:29.263958 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:29.263935 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/3.log" Oct 30 01:43:31.441613 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:31.441582 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-xjbn5_3bc7448f-822d-4574-af12-0545a3513234/volume-data-source-validator/0.log" Oct 30 01:43:34.010487 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:34.010454 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xwtsr_70d2e011-58cb-4646-adb9-04f8eacab25c/dns/0.log" Oct 30 01:43:34.113974 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:34.113941 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xwtsr_70d2e011-58cb-4646-adb9-04f8eacab25c/kube-rbac-proxy/0.log" Oct 30 01:43:34.436728 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:34.436650 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnlzg_dd14a22e-0b4d-45eb-a25e-111817cfe34d/dns-node-resolver/0.log" Oct 30 01:43:34.640082 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:34.640051 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d54frhj_9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a/istio-proxy/0.log" Oct 30 01:43:35.548049 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:35.547993 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2j4gq_90aa2cb7-11fc-4a3e-8bad-519c2c0abcbb/node-ca/0.log" Oct 30 01:43:36.660812 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:36.660782 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:43:36.661494 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:36.661470 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lzlc8_fdd0dfbe-2086-4370-8336-051395569bd8/console-operator/2.log" Oct 30 01:43:37.489253 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:37.489219 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d54frhj_9b9e4cb9-5297-40fb-aa9c-ec8bf348bc6a/istio-proxy/0.log" Oct 30 01:43:39.169741 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:39.169715 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8rf4k_26e2681e-21d4-42a6-967c-d462361c99e6/discovery/0.log" Oct 30 01:43:39.203811 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:39.203783 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7597b65cbf-jb59g_f9305cf4-d3b1-43cc-b248-33a8d2bbca5c/kube-auth-proxy/0.log" Oct 30 01:43:39.333394 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:39.333363 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-2rs7m_2e721927-370e-4f47-9eef-60330ef5886a/istio-proxy/0.log" Oct 30 01:43:39.403362 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:39.403330 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-2z245_b1988474-6586-4686-b98d-34a146461f5e/istio-proxy/0.log" Oct 30 01:43:39.585414 ip-10-0-130-171 kubenswrapper[4905]: I1030 01:43:39.585391 4905 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hzfrs_f9b68d31-1fc7-4045-9167-11e44e372333/istio-proxy/0.log"